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1.  INTRODUCTION 

The  rapid  advance  of  information  systems 
technology  and  the  broader  spectrum  of  threats 
we  face  has  caused  our  Command,  Control, 
Communications,  Computers,  Intelligence, 
Surveillance  and  Reconnaissance  (C4ISR) 
procedures  and  architectures  to  undergo 
profound  changes.  Joint  Vision  2010  highlights 
"Information  Superiority"  as  the  major  enabler 
for  achieving  Full  Spectram  Dominance  and 
C4ISR  is  the  major  contributor  to  achieving 
information  superiority.  This  notwithstanding, 
the  resource  constrained  environment  of  the 
Department  of  Defense  (DoD)  is  forcing 
decision  makers  to  face  tough  choices  in 
determining  the  allocation  of  investment 
between  C4ISR  and  the  other  contributors  to 
force  effectiveness.  Cogent  analyses  are  needed 
to  support  these  decisions. 

Heretofore,  the  analysis  community  has  not 
effectively  been  able  to  assess  the  relative 
contribution  of  C4ISR  to  force  effectiveness  vis- 
a-vis  other  factors.  Past  analyses  have  not  been 
adequate  for  the  new  emphasis  on  C4ISR.  The 
analytic  community  must  now  posture  itself  to 
address  the  analysis  of  C4ISR  for  2010  and 
beyond. 

The  Military  Operations  Research  Society 
(MORS)  organized  and  conducted  a  workshop 
to  examine  the  question  of  Analyzing  C4ISRfor 
2010.  That  workshop  was  held  27-29  October 
1998  at  the  Center  for  Strategic  Leadership,  US 
Army  War  College,  Carlisle  Barracks,  PA.  The 
General  Chair  of  the  Workshop  was  RADM 
Robert  Nutwell,  USN.  Captain  Jay  Kistler, 

USN,  was  the  Deputy  Chair  and  Dr.  Russell 
Richards,  MITRE  Corporation,  was  the 
Technical  Chair. 

LI  Goals  and  Objectives  of  the  Workshop.  The 
goals  of  the  workshop  were  to: 

1.  Share  information  on  the  current  state  of 
C4ISR  analysis. 

2.  Determine  the  appropriate  metrics  and 
methodologies  for  analysis  of  C4ISR  for 
2010. 


3.  Define  key  areas  of  improvement  for  the 
application  of  analysis  in  support  of  decision 
making  on  C4ISR  investments  and 
warfighting  utility. 

The  objectives  were  to: 

•  Enumerate  the  relevant  issues  pertaining  to 
the  analysis  of  C4ISR  in  2010. 

•  Identify  metrics  that  are  sensitive  to  the 
effects  of  C4ISR  on  force-level 
effectiveness. 

•  Assess  methodologies  to  analyze  and 
quantify  the  effectiveness  of  C4ISR 

•  Evaluate  appropriate  tools  to  measure  the 

•  benefit  of  C4ISR. 

•  Discuss  the  requirements  for  and 
employment  of  advanced  tools,  methods, 
and  research. 

1.2  Approach.  The  subject  of  C4ISR  analysis 
was  examined  in  the  context  of  a  number  of 
different  mission  areas.  A  working  group  was 
assigned  to  each  mission  area.  The  mission 
areas  were: 

1.  Major  Theater  War. 

2.  Smaller  Scale  Contingencies. 

3.  Operations  Other  Than  War. 

4.  Infrastracture  Assurance. 

5.  Peacetime  Engagement. 

In  addition,  separate  working  groups  were 
assigned  to  the  focus  areas  of  Information 
Architectures  and  Analytical  Techniques  and 
Tools. 

1.3  Taskings.  Consistent  with  the  goals  and 
objectives  of  the  workshop,  each  working  group 
was  charged  to  undertake  five  taskings: 

1.  Characterize  C4ISR  within  the  working 
group  mission  area  or  focus  area. 

2.  Define  the  relative  worth  of  C4ISR. 

3.  Develop  and  recommend  Measures  of  Merit 
(MoM). 

4.  Identify  and  describe  tools. 

5.  Analyze,  synthesize  and  infer. 
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2.  BACKGROUND 

The  increasing  emphasis  on  joint  operations 
driven  by  the  Nichols/Goldwater  legislation,  the 
information  age  and  the  changing  military  threat 
have  led  to  significant  changes  in  the 
Department  of  Defense.  The  Joint  Warfighting 
Capability  Assessments  (JWCAs)  are  performed 
for  each  of  1 1  joint  warfighting  capability  areas. 
Reflecting  the  emphasis  on  conducting  joint 
operations,  all  of  the  warfighting  capability  areas 
cut  across  service  boundaries.  With  most  of  the 
analysis  tools  having  been  developed  by  a  single 
service  to  support  the  specific  needs  of  that 
service,  the  tools  available  to  support  joint 
analysis  and  to  support  tradeoff  decisions  which 
cross  service  boundaries  are  deficient.  This  is 
particularly  true  for  evaluating  the  contributions 
ofC4ISR. 

Recognizing  these  deficiencies,  the  Vice 
Chairman  of  the  Joint  Chiefs  of  Staff  and 
Chairman  of  the  Joint  Requirements  Oversight 
Council  (JROC)  in  1995,  Admiral  William 
Owens,  asked  the  Military  Operations  Research 
Society  to  host  a  workshop  to  look  at  the  ability 
of  the  community  to  support  the  JWCAs.  That 
workshop  was  conducted  October  1995.  It  had 
working  groups  for  each  of  the  eleven  joint 
warfighting  capability  areas.  Each  working 
group  evaluated  the  "health"  of  the  analysis 
capabilities  as  they  pertain  to  the  capability  area. 
Most  reported  serious  deficiencies  in  the  models, 
simulations,  databases,  measures  and 
methodologies.  This  was  particularly  true  for 
the  C2IW  and  ISR  working  groups.  Influenced 
significantly  by  that  discouraging  evaluation  of 
the  tools.  Admiral  Owens  and  other  senior 
decision  makers  in  the  Department  of  Defense 
took  steps  to  remedy  the  situation. 

Two  manifestations  of  the  actions  taken  to 
remedy  the  problem  were  the  initiation  of  the 
Joint  Analytic  Model  Improvement  Program 
(JAMIP)  and  the  C4ISR  Decision  Support  Task 
Force  (DSTF).  The  JWARS  simulation 
development  effort  and  the  Joint  Data  System 
(JDS)  program  for  improving  the  databases  for 
supporting  joint  analysis  were  two  programs  that 


evolved  from  JAMP.  The  DSTF  concluded  that 
the  existing  tools  were  severely  deficient  for 
meeting  the  C4ISR  analysis  requirements.  It 
also  concluded  that  DoD  needed  an  organization 
focusing  specifically  on  C4ISR  analysis.  That 
task  force  led  to  the  creation  of  the  C4ISR 
Decision  Support  Center  (DSC)  and  the  C4ISR 
Joint  Battle  Center  (JBC). 

In  the  few  years  following  the  DSTF,  there  has 
been  continuing  interest  and  emphasis  on  C4ISR 
analysis.  There  has  been  much  work  towards 
improving  the  capabilities  of  the  tools  (e.g., 
JWARS,  C4ISR  Federation,  NETWARS),  and 
there  have  been  several  major  C4ISR  analysis 
efforts  including  the  C4ISR  Mission  Assessment 
(CMA)  conducted  to  support  the  Quadrennial 
Defense  Review  (QDR).  Nevertheless,  there  are 
still  many  problems  with  assessing  the 
contribution  of  C4ISR  to  warfighter 
effectiveness  and  to  trade-off  the  contributions 
of  C4ISR  with  other  systems.  This  MORS 
workshop  provided  an  opportunity  to  reassess 
the  situation  and  to  offer  recommendations  to 
our  DoD  Sponsors  about  improving  the  health  of 
C4ISR  analysis. 

3.  THE  WORKSHOP  ACTIVITIES 

3.1  Plenary  Sessions.  The  workshop  kicked  off 
the  first  day  with  four  excellent  presentations, 
each  of  which  validated  the  need  for  our  meeting 
and  which  offered  challenges  to  the  participants. 

“The  Decision  Maker's  Perspective.”  Mr.  Art 
Money,  the  Assistant  Secretary  of  Defense  for 
Command,  Control,  Communications  and 
Intelligence  (ASD(C3I))  and  the  Department  of 
Defense’s  Chief  Information  Officer,  started 
things  off  with  a  keynote  address  providing  the 
decision  maker’s  perspective.  He  emphasized 
the  importance  of  information  superiority  and 
acknowledged  that  the  topics  and  goals  of  our 
workshop  were  timely  and  on  track.  His  talk 
provided  an  interesting  historical  perspective  on 
conflicts  involving  the  United  States.  Mr. 
Money  considered  the  six  US  security  epochs 
listed  in  Table  1. 
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Table  1.  Epochs  in  US  Security  History 


Time  Period 

Description 

To  1820 

Revolutionary  War 

2 

1820-1900 

Civil  War 

3 

1900-1920 

WWr.  Birth  of  Communism 

4 

1920-1945 

WWII:  Fascism  defeated,  atomic  bomb,  massive  technology  advances 
(aircraft,  communications,  radars,  UHF,  etc.) 

5 

1945-1993 

Cold  War:  monolithic  threat  (Soviet  Union),  Vietnam,  Korea,  ICBM,  Fulda 
Gap,  SAC;  predictability 

6 

1993-1998 

Coalition  War:  collapse  of  a  major  power,  walls  come  down,  transnational 
threat,  non-government  powers,  no  geographic  boundaries,  asymmetric 
warfare,  critical  infrastructure  protection,  information  warfare,  sensitivities 
of  information,  operations  other  than  war 

The  first  epoch  covered  the  establishment  of  the 
US  as  a  nation  that  was  pursuing  an  isolationist 
foreign  policy.  This  isolationism  continued 
through  the  second  epoch,  from  1820-1900, 
when  the  US  was  involved  with  the  Civil  War 
and  later  becoming  a  world  power.  The  third 
epoch,  from  1900-1920,  was  characterized  by 
World  War  I,  the  collapse  of  Europe  and  the 
birth  of  communism.  The  fourth  epoch,  from 
1920-1945,  witnessed  massive  technological 
advancement,  and  included  World  War  H,  the 
defeat  of  Fascism  and  the  development  of  the 
atomic  bomb.  Epoch  number  five,  from  1945- 
1990,  includes  the  Korean  War,  the  Vietnamese 
War,  the  Cold  War,  the  eerie  predictability  of  a 
monolithic  threat  and  bi-polar  powers. 

The  sixth  security  epoch,  from  1990  to  the 
current  time,  presents  a  new  paradigm.  The 
collapse  of  the  Soviet  Union  as  a  world  power,  a 
major  desert  war  and  the  emergence  of  many 
turbulent  forces  have  highlighted  it.  These 
include  the  500-700  year  old  gradges  between 
often  well  armed  nation  states,  technological 
advancement  and  commercial  expansion,  the 
proliferation  of  WMDs,  and  threats  to  the  US 
that  include  quasi-govemmental  organizations 
(Bin  Laden),  transnational  entities  and  tribes. 
The  threats  are  more  asymmetric  than  prior 
threats  and  include  threats  to  the  US 
infirastructure  and  cyber  attacks. 

This  perspective  demonstrated  the  shift  in  the 
defense  requirements  that  has  taken  place  during 


our  country’s  history  and  demonstrated  why 
there  is  increasing  interest  in  C4ISR. 

Mr.  Money  emphasized  the  importance  of 
information  superiority  —  providing  the  right 
information  to  the  right  people  in  the  right 
format  at  the  right  time.  However,  he  also 
emphasized  the  need  to  assess  the  Return  On 
Investment  (ROI)  for  C4ISR  systems  — 
precisely  the  reason  for  the  MORS  workshop. 

He  challenged  the  workshop  to  determine  what  a 
pound  ofC4ISR  is  worth.  He  stated  that  we  do 
not  presently  do  this  well,  and  we  need  better 
tools  and  methodologies  to  evaluate  the 
contribution  that  C4ISR  makes  to  force 
effectiveness.  He  specifically  highlighted  the 
need  for  better  models  and  simulations  and  for 
better  measures. 

3.1.2  “The  Operator’s  Perspective.”  Mr. 
Money’s  talk  was  followed  by  two  presentations 
providing  the  operator’s  perspective.  The  first 
was  given  by  Brig  Gen  Carol  Elliott,  the  Vice 
Director  for  the  Joint  Staff  J-2  and  the  Team 
Lead  for  the  Intelligence,  Surveillance  and 
Reconnaissance  (ISR)  Joint  Warfighting 
Capability  Assessment  (JWCA).  She  was  a 
voice  of  experience,  for  she  has  been  grappling 
with  the  problem  of  determining  the  ROI  for 
ISR  during  the  last  couple  of  years,  most 
recently  the  Recce  2010  Study.  She  echoed  the 
remarks  of  Mr.  Money  in  indicating  that  the 
analysis  community  faces  big  challenges  in 
determining  the  ROI  for  ISR,  and  in 
acknowledging  that  we  do  not  do  this  very  well. 
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She  stated  that  the  JWCA  process  has  a 
voracious  appetite  for  people  who  can  bring 
coherent  analysis  to  the  table.  She  stated  that 
analysis  of  the  value  of  C4ISR  would  require 
innovative,  new  approaches. 

General  Elliott  gave  a  pithy  discussion  of 
analysis  pitfalls  derived  from  her  own 
experiences  in  supporting  the  ISR  JWCA.  Some 
of  the  points  that  she  made  were: 

•  Make  sure  that  terms  such  as  "Dominant 
Battlespace  Awareness"  are  well  defined. 

•  Make  sure  that  all  analysis  assumptions  are 
made  clear  to  the  decision  makers. 

•  The  use  of  notional  data  is  a  real  problem. 

•  Be  sure  the  analysis  does  not  exclude  people 
as  part  of  the  ISR  process. 

•  Be  sure  that  the  measures  of  merit  are 
relevant  (it  is  not  just  a  problem  of 
mathematics). 

•  It  is  always  difficult  to  determine  how  much 
analysis  is  enough. 

•  Whatever  the  results,  you  will  be  misquoted, 
misinterpreted,  attacked,  disregarded  or  all 
of  the  above. 

•  Keep  the  "I"  in  ISR.  It  is  more  than  a 
question  of  platforms. 

•  Models  are  not  adequate  for  the  types  of 
tradeoff  analyses  that  they  are  being  asked 
to  do.  Models  used  for  ISR  analyses  were 
developed  for  other  things.  Better  measures 
are  needed  to  assess  the  contribution  of  ISR. 

General  Elliott  closed  with  five 
recommendations  for  C4ISR  analysis: 

1.  Define  effectiveness. 

2.  Replace  assertions  with  facts. 

3.  Scrutinize  results  for  what’s  missing. 

4.  Explain  your  results  to  your  administrative 
staff  to  see  if  your  explanation  passes  the 
common  sense  test. 

5.  Remember  that  it’s  people,  stupid! 

Colonel  William  Cooper,  representing  the  Joint 
Staff  J-6,  provided  the  second  operator’s 
perspective.  Like  General  Elliott,  Colonel 
Cooper  is  a  JWCA  team  leader  -  for  the 
Command  and  Control  area.  Colonel  Cooper’s 


presentation  focused  on  the  problems  with  the 
legacy  stovepiped  tools  for  addressing  C2  issues. 
He  discussed  the  need  for  service  models  to 
interoperate  so  that  joint  issues  can  be 
addressed.  Colonel  Cooper  also  stressed  the 
need  for  more  than  M«feS  to  support  C4ISR 
analyses.  He  discussed  the  need  to  coordinate 
joint  experimentation  activities  of  USACOM  J-9 
with  the  modeling  and  simulation.  He  saw  the 
opportunity  to  validate  models  with 
experiments.  He  closed  by  challenging  the 
community  to  develop  better  models  for  C4ISR 
and  to  tie  weapons  systems  and  C4ISR  together. 

3.1.3  Technical  Perspective  —  “C2 
Assessment:  Past,  Present  and  Future.”  Dr. 
Stuart  Starr,  MITRE  Corporation,  completed  the 
day’s  invited  presentations  with  a  historical 
perspective  on  the  changes  that  he  has  seen  over 
the  past  25  years  with  respect  to  command  and 
control  assessments.  Dr.  Starr  discussed  three 
“ages”: 

•  Awakening 

•  Dark  Ages 

•  Renaissance 

Prior  to  1975,  there  was  general  insensitivity  to 
C2  issues.  C2  was  often  ignored  or  assumed  to 
be  perfect  or,  if  considered,  it  was  often  treated 
simply  as  a  patch  (e.g.,  an  additional  term  in 
Lanchester’s  equations).  The  little  bit  of 
analysis  of  C2  that  was  done  was  fragmented 
within  the  Department  of  Defense.  The  decade 
from  1975—1985,  ths  Awakening,  saw  an 
increased  interest  by  the  senior  leadership  in  C3I 
leadership  as  there  was  an  increased  awareness 
of  the  importance  of  C3I  and  a  need  to  justify 
expenditures  on  C3I.  This  increased  interest 
was  manifest  through  several  key  initiatives 
which  included  symposia  and  workshops  (at 
least  one  sponsored  by  MORS  on  C3I  Measures 
of  Effectiveness),  the  creation  of  new 
organizations  (e.g.,  the  C2  Research  Program 
and  the  Naval  Postgraduate  School  C3 
Curriculum),  major  multi-year  studies  focusing 
on  C3I  and  increasing  resources  devoted  to  C3I 
assessment.  New  assessment  methodologies  and 
processes  such  as  the  Modular  C2  Evaluation 
Structure  (MCES),  Headquarters  Effectiveness 
Assessment  Tool  (HEAT),  Mission  Oriented 
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Approach  (MOA)  and  new  tools  such  as 
synthetic  environments  like  the  Identify  Friend 
or  Foe  Network  (IFFN)  Testbed  greatly 
improved  the  ability  to  assess  the  effectiveness 
of  C3I.  Much  of  the  credit  for  the  Awakening 
was  due  to  the  strong  leadership  of  people  like 
Robert  Hermann  and  Harry  Van  Trees. 

The  Awakening  period  was  followed  by  a  six 
year  period  from  1985-1990,  the  Dark  Ages,  in 
which  DoD  leadership  showed  a  significant 
reduction  in  interest  in  C3I  assessments.  This 
was  due  to  relatively  high  budgets  for  C3I  and 
an  emphasis  on  C3I  system  acquisition. 

Because  resources  were  plentiful,  there  was  little 
concern  about  return  on  investment  for  C3I.  The 
influence  of  analysis  organizations  waned 
significantly  during  the  Dark  Ages. 

The  Dark  Ages  were  followed  by  the 
Renaissance  period  (1991-Present).  This  was 
brought  about  by  a  combination  of  factors  which 
included  profound  geopolitical  changes  (e.g.,  the 
dissolution  of  the  USSR  and  the  Warsaw  Pact), 
severe  resource  reductions  and  changing 
paradigms  such  as  the  Revolution  in  Military 
Affairs  (RMA)  and  Simulation  Based 
Acquisition  (SBA).  Some  of  the  manifestations 
of  the  renewed  interest  in  analysis  and  C3I 
assessments  were  the  creation  of  the  Defense 
Modeling  and  Simulation  Office,  the 
revitalization  of  the  CCRP,  the  C4ISR  Decision 
Support  Task  Force,  the  creation  of  the  C4ISR 
Decision  Support  Center  (DSC)  and  the  Joint 
C4ISR  Battle  Center  (JBC),  the  C4ISR  Mission 
Assessment  (CMA),  the  initiation  of  the  Joint 
Analytic  Model  Improvement  Program 
(JAMIP),  the  development  of  major  analysis 
simulation  models  which  made  major 
improvements  in  the  representation  of  C4ISR 
(e.g.,  JWARS,  EAGLE,  ARES,  C4ISR  Model 
and  NSS)  and  the  establishment  of  the  Joint 
Experimentation  Directorate  within  the  US 
Atlantic  Command  (USACOM).  MORS  has 
played  a  major  role  in  the  Renaissance  in  C4ISR 


assessment.  The  following  MORS-sponsored 
workshops  have  contributed  to  the  renaissance: 

•  C3IEW  MOEs 

•  W&AforM&S 

•  JROC  Analysis 

•  Advanced  Distributed  Simulation  for 
Analysis 

•  OOTW  Analysis  Methods  and  Techniques 

•  Quick  Response  Analysis  Methodologies 
(QRAM) 

•  Warfare  Analysis  and  Complexity  (“New 
Sciences”) 

•  QDR  Lessons  Learned 

•  SIMTECH  2007 

Two  NATO  Panels  made  significant 
contributions  to  the  renaissance  during  this 
period.  The  NATO  Panel  7  Working  Group 
conducted  several  meetings  aimed  at  assessing 
the  impact  of  C3I  on  the  battlefield.  The  NATO 
Studies,  Analysis  and  Simulations  (SAS)  Panel 
published  a  Code  of  Best  Practices  (COBP)  for 
Assessing  C2  in  1998.  The  COBP 
recommended  an  Assessment  Methodology, 
which  consisted  of  the  following  elements: 

•  Problem  Structuring 

•  Human  Factors  and  Organizational  Issues 

•  Scenarios 

•  Measures  of  Merit 

•  Tools  and  Their  Applications 

•  Data 

•  Risk  and  Uncertainty 

•  Reporting 

Dr.  Starr  then  referenced  General  Larry  Welch 
in  making  the  case  that  DoD  leadership  must 
initiate  and  sustain  some  major  cultural  changes 
to  track  the  other  changes  that  have  occurred  if 
our  C4ISR  analyses  are  to  be  relevant.  Table  2 
describes  the  areas  in  which  cultural  changes  are 
needed  and  the  nature  of  the  changes. 
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Table  2.  Needed  Cultural  Changes 


Area 

Sub-Area 

From 

To 

Institutional 

Outcomes 

Protected/ Advocacy 

Open/Unbiased 

ReviewA^V&A 

Bureaucratic 

Subject  matter  expert 

Organizational 

Stovepiped 

Collaborative 

M&S 

Orientation 

Model 

Subject  matter 

Processes 

Opaque 

Transparent 

Data 

Limited  to  “validated” 

Include  possibilities 

Algorithms 

Stable,  traceable 

Allow  for  non-linear 

Mission  Orientation 

Cold  War 

New  World  Disorder 

Processes 

Uncertainty 

Suppress 

Illuminate 

Scenarios 

Few,  “blessed” 

Full  range 

Analysis 

Force-on-force 

Military-social-economic 

Forces,  concepts 

Symmetrical 

Asymmetrical 

Trade-offs 

Limited 

Full  range 

Scope 

Narrow  (force  structure; 
military  worth  of 
equipment) 

Broad  (Forces; 
equipment;  doctrine; 
concepts;  C2; ...) 

Dr.  Starr  closed  his  presentation  with  a 
discussion  of  selected  reconunendations  from 
the  COBP,  a  characterization  of  the  C2 
assessment  base  and  some  major  challenges. 
Since  many  of  his  recommendations  and 
challenges  reappeared  in  some  of  the  briefouts, 
we  will  fold  in  his  recommendations  and 
challenges  with  those  of  the  working  groups 
later  in  this  report. 

3.1.4  ^‘Federated  Analysis  of  C4ISR  in 
Warfare.”  On  the  second  day,  Mr.  Charles 
Taylor,  C4ISR  Decision  Support  Center,  gave  an 
interesting  luncheon  presentation  sharing  some 
of  his  recent  experiences  in  grappling  with 
C4ISR  analysis  problems  at  the  DSC.  Mr. 
Taylor  made  a  strong  point  that  no  single  model 
will  be  adequate  to  meet  the  needs  of  the  C4ISR 
analysis  community.  He  advocated  the  use  of  a 
federation  of  M&S  tools  to  support  the  analysis. 

3.1.5  “Blackhawk  Down.”  Mr.  Mark  Bowden, 
a  reporter  for  the  Philadelphia  Enquirer,  gave  a 
luncheon  presentation  on  the  third  day.  His 
multi-media  presentation  described  the  gripping 
real-life  activities  surrounding  the  attempt  to 
rescue  the  US  crew  of  a  helicopter  that  crashed 
in  Mogadishu.  His  presentation  highlighted 


some  of  the  critical  problems  with  our  C4ISR 
systems. 

3.2  Conduct  of  the  Working  Groups.  After  the 
first  day’s  plenary  session,  the  participants  broke 
into  their  respective  working  groups.  In  addition 
to  addressing  the  taskings  of  the  Workshop,  each 
working  group  was  instructed  to  evaluate  the 
health  (in  the  mission  area)  of  the  state  of  C4ISR 
analysis. 

3.3  Synthesis  Team.  As  has  become  customary 
for  MORS  workshops,  a  Synthesis  Team  was 
created  to  provide  a  holistic  view  of  the 
deliberations  of  all  of  the  working  groups 
participating  in  the  workshop.  Consistent  with 
that  goal,  the  team  included  representatives  from 
OSD,  each  of  the  services,  industry,  FFRDCs 
and  academia.  Selected  members  of  the 
Synthesis  Team  actively  participated  in  each  of 
the  working  groups  to  help  the  groups  keep  on 
track  with  respect  to  meeting  the  objectives  of 
the  workshop  and  to  report  on  progress  or 
difficulties  encountered  by  the  groups.  The 
Synthesis  Team  convened  periodically  during 
the  course  of  the  workshop  to  discuss  and 
compare  the  evolving  findings  and 
recommendations  of  each  working  group. 
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3.3.1  Products  of  the  Synthesis  Team.  The 
Synthesis  Team  drew  on  its  periodic 
deliberations  and  its  observations  of  the 
activities  in  the  working  groups  to  develop  some 
broad  observations  about  the  nature  of  the 
problem  and  to  formulate  a  set  of  cross-cutting 
findings  and  recommendations.  The  products  of 
the  Synthesis  Team  are  summarized  below. 

3.3.2  Broad  Observations.  As  a  consequence  of 
monitoring  the  deliberations  of  the  individual 
mission-oriented  working  groups  it  was  clear 
that  there  was  broad  variability  in  the  maturity 
of  the  individual  mission  areas  (e.g.,  level  of 
domain  knowledge;  available  tools  and  data; 
experience  in  analyzing  the  C4ISR  systems  and 
processes  associated  with  those  missions). 

Based  on  the  perceptions  of  the  Synthesis  Team, 
it  was  concluded  that  one  would  rank  the 
missions  on  a  qualitative  “maturity  scale”  as 
follows  (most  mature  to  least  mature); 

1 .  Major  Theater  War  (MTW) 

2.  Smaller  Scale  Contingency  (SSC) 

3.  Operations  Other  Than  War  (OOTW) 

4.  Peacetime  Engagement  (PE) 

5.  Infrastructure  Assurance  (lA) 

This  maturity  ranking  should  not  be  surprising 
as  it  tracks  with  the  total  amount  of  effort 
devoted  to  C4ISR  analysis  for  the  different 
mission  areas. 

The  Synthesis  Team  further  observed  that  many 
of  these  mission  areas  subsume  qualitatively 


different  sub-missions.  For  example.  Operations 
Other  Than  War  (OOTW)  subsumes 
Humanitarian  Assistance  (HA),  disaster  relief, 
peace  keeping  and  peace  enforcement.  These 
sub-missions  are  so  different  in  their  nature  and 
maturity  that  it  can  be  misleading  to  making 
overarching  statements  about  the  overall  mission 
area.  For  example,  some  of  the  sub-missions  in 
OOTW  would  be  rated  very  immature. 

The  Synthesis  Team  also  noted  problems  with 
definition  for  several  of  the  mission  areas. 
Notably,  the  distinction  between  SSC  and 
OOTW  was  not  initially  clear.  To  eliminate 
confusion,  the  chairs  of  these  two  working 
groups  “defined”  their  areas  of  focus  to  make 
the  distinction.  Furthermore,  the  Peacetime 
Engagement  Working  Group  spent  a  significant 
amount  of  time  crafting  a  definition  of  PE. 
Finally,  because  infrastructure  assurance  was  a 
relatively  new  mission  area,  it  was  necessary  for 
the  co-chairs  to  clearly  define  lA. 

3.3.3  Major  Issues.  As  the  Synthesis  Team 
monitored  the  deliberations  of  the  individual 
working  groups,  it  identified  several  major 
issues  that  were  raised  by  those  working  groups. 
The  major  issues  are  summarized  in  Table  3. 
Note  that,  although  there  are  considerable 
differences  in  those  issues,  several  crosscutting 
issues  are  apparent.  These  crosscutting  issues 
involve  difficulties  associated  with  data  and 
Measures  of  Merit  (MoMs).  These  specific 
crosscutting  issues  are  discussed  in  the  findings 
and  recommendations  that  follow. 


Table  3:  Major  Issues 


Mission  Area 

Major  Issues 

MTW 

Few  accepted  data  sources;  Modeling  Red  C4ISR; 

Existing  data  does  not  map  to  DPG  scenarios 

SSC 

MoMs  need  sufficient  pre-definition 

OOTW 

Data;  Softness  of  MoMs  (e.g.,  public  opinion); 

OOTWs  are  not  all  the  same 

Peacetime  Engagement 

Definition  of  Peacetime  Engagement 

Infrastructure  Assurance 

Data  (much  of  it  is  Blue  data);  Higher-level  MoMs; 

Scope  of  problem/ability  to  decompose  meaningfully 

Dealing  with  the  interactive  nature  of  the  problem 

Information  Architectures 

How  does  one  score  an  architecture  and  communicate  the  results  appropriately? 

Analytical  Tools  and 

Cognitive  modeling;  Broad  array  of  missions; 

Techniques 

Size  of  the  trade  space:  Representation  of  Red  data 
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Each  working  group  assessed  the  current  state  of 
the  health  of  C4ISR  analysis  in  its  mission  area. 
The  template  that  was  developed  for  the 
forthcoming  NATO  COBP  for  Assessing 
Command  and  Control  (and  which  was 
introduced  by  Dr.  Stuart  Starr  in  his  plenary 
presentation)  was  used  for  the  working  group 
assessments.  Table  4  provides  an  ‘aggregate’ 
stop  light  assessment  (Green  =  good;  Yellow  = 
fair;  Orange  =  significantly  deficient;  Red  = 
poor),  aggregated  qualitatively  across  the  results 
of  the  individual  working  group  assessments. 


Table  4.  Aggregate  Assessment  of  the  Health  of 
C4ISR  Analysis 


Element 

Aggregate  Assessment 

Problem  Structuring 

Orange 

Human  Factors/ 
Organization 

Orange 

Scenarios 

Yellow 

Measures 

Orange 

Data 

Red 

Tools 

Red 

Risk  and  Uncertainty 

Orange 

Report 

Orange 

Note  that  none  of  the  elements  were  rated  Green. 
Also,  two  of  the  critical  steps  (Data  and  Tools) 
were  assessed  as  being  Red,  and  five  of  the  steps 
(Problem  Structuring,  Human 
Factors/Organization,  Measures  of  Merit, 
Treatment  of  Risk  and  Uncertainty  and  Report 
Documentation  and  Availability)  were  assessed 
as  being  Orange.  Only  the  Scenario  Generation 
step  was  rated  as  good  as  Yellow. 

These  observations  suggest  concern  about  the 
ability  of  the  analysis  community  to  adhere  to 
the  processes  in  a  strawman  COBP.  These 
results  led  the  Synthesis  Team  to  highlight 
rindings  and  recommendations  in  five  areas; 
COBP,  three  key  elements  of  the  candidate 
COBP  and  the  overall  relationship  between  the 
producer  and  consumer  of  C2  anidyses. 

3.3.4  Findings.  The  findings  of  the  Synthesis 
Group  are  discussed  below. 

COBP.  The  NATO  COBP  provides  an 
interesting  and  useful  first  step  towards 


developing  a  sound  foundation  for  C4ISR 
analysis.  However,  the  scope  of  the  COBP  was 
restricted  to  C4ISR  analysis  in  the  context  of 
conventional  warfare.  As  this  workshop 
revealed,  the  problem  of  analyzing  C4ISR  for 
2010  will  involve  a  host  of  additional  mission 
areas,  which  pose  unique  problems  for  the 
C4ISR  assessment  community.  Thus,  an 
expanded,  community-endorsed  COBP  is 
needed  to  support  the  C4ISR  assessment 
problem,  in  all  its  dimensions. 

Measures  of  Merit  (MoMs).  Over  the  last 
thirteen  years,  MORS  has  conducted  several 
workshops  on  the  issue  of  developing  measures 
of  merit  for  command  and  control  assessments  [ 
Ref.  1,2].  Those  workshops  proposed  the 
concept  of  a  hierarchy  of  MoMs  that  would 
range  from  measures  of  system  performance 
through  measures  of  mission  effectiveness.  The 
various  mission-oriented  working  groups  at  this 
workshop  appeared  to  support  this  conceptual 
approach  to  the  problem.  However,  it  was 
concluded  that  the  hierarchy  must  be  extended  to 
deal  with  the  needs  of  new  world  disorder 
missions.  The  Architecture  Working  Group  also 
observed  that  it  lacked  MoMs  to  support  the 
effective  assessment  and  comparison  of 
architectural  options.  In  particular,  it  called  for 
MoMs  that  would  relate  system  architectural 
characteristics  to  operational  outcomes. 

Data.  A  decade  ago,  at  a  MORS  Workshop  on 
Simulation  Technology  1997  (SIMTECH  97), 
Dr.  Walt  LaBerge  observed  that  “without  data 
we  are  nothing”  [Ref.  3].  That  thought  clearly 
prevailed  in  this  workshop.  Each  working  group 
highlighted  data  deficiencies.  The  specific 
concerns  varied  by  Working  Group  and  included 
the  following  challenges: 

1 .  Acquiring  the  Blue  system  data  needed  to 
perform  vulnerability  assessments  of  critical 
infrastructiures. 

2.  Acquiring  the  data  appropriate  to  some 
currently  ill-defined  future  military 
operation. 

3.  Characterizing  Red’s  C4ISR  systems  and 
critical  infrastructures. 

4.  Gaining  access  to  needed  data  in  a  timely 
fashion. 
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5 .  Collecting  and  making  better  use  of  data 
emerging  from  Service  and  Joint  Advanced 
Warfighting  Experiments. 

6.  Validation  and  certification  of  data. 

Tools.  Many  of  the  speakers  at  the  workshop 
focused  their  remarks  on  modeling  and 
simulation  and  its  important  role  in  C4ISR 
analysis.  It  is  clear,  however,  that  no  single 
class  of  tool  (and  certainly  no  single  tool  within 
a  class)  can  satisfy  C4ISR  assessment  needs 
adequately.  There  is  a  broad  spectrum  of  tools 
that  a  C4ISR  analyst  can  choose  from,  that  vary 
with  respect  to  the  time  to  create  and  apply  the 
tools,  the  cost  to  create  and  apply  them,  the 
amount  of  abstraction  and  their  credibility  (see 
Ref  [4]). 

Relations  between  Providers  and  Consumers 
of  C4ISR  Assessments.  Many  Working  Groups 
observed  that  the  relationships  between  the 
providers  and  the  consumers  of  C4ISR 
assessments  are  strained.  By  relying  on 
complex,  opaque  M&S  tools,  many  providers 
are  failing  to  make  their  analyses  transparent  and 
understandable  (e.g.,  “Why  did  I  come  to  that 
conclusion?  My  model  told  me!”).  Similarly, 
consumers  are  frequently  not  “educated 
customers.”  For  example,  they  fail  to  articulate 
issues  in  a  way  that  is  amenable  to  responsible 
C4ISR  analysis.  Or,  they  fail  to  give  the 
provider  adequate  resources  to  either  create  the 
needed  tools  or  to  perform  the  assessment. 

3.3.5  Recommendations.  This  section  provides 
recommendations,  that  the  Synthesis  Group  felt 
would  improve  the  state  of  C4ISR  analysis. 

Each  of  the  listed  findings  (Section  3.3.4)  are 
addressed. 

COBP.  Although  the  preliminary  review  of  the 
NATO  COBP  was  promising,  we  recommend 
that  a  more  thorough,  in-depth  assessment  of  the 
product  should  be  undertaken  to  see  if  it  meets 
(at  least  some  of)  the  needs  of  the  US  C4ISR 
analysis  community.  MORS  is  in  the  best 
position  of  any  US  organization  to  organize  and 
execute  such  an  assessment. 

If  such  an  assessment  is  undertaken  and  it 
endorses  the  NATO  COBP  as  a  preliminary 


product,  the  following  steps  should  be  pursued. 
First,  the  product  should  be  disseminated 
broadly  to  the  C4ISR  assessment  community, 
along  with  any  caveats  that  emerged  through  the 
assessment  process.  MORS  has  an  efficient 
distribution  network  for  making  such 
information  broadly  available.  Second, 
consideration  should  be  given  to  employing  the 
product  in  defense  universities,  such  as  the 
Naval  Postgraduate  School  (NFS)  and  the  Air 
Force  Institute  of  Technology  (AFIT),  to  educate 
young  analysts. 

In  view  of  the  limited  scope  of  the  NATO 
COBP,  efforts  should  be  supported  to  extend  the 
product  to  address  new  world  disorder  issues. 
NATO  is  forming  Studies,  Analysis  and 
Simulation  (SAS)  Panel-015,  to  explore  the 
feasibility  of  extending  the  initial  COBP  to  deal 
with  issues  associated  with  OOTW  and 
information  operations.  Those  efforts  should  be 
supported  by  MORS  (e.g.,  by  providing 
representatives  to  the  deliberations;  appropriate 
briefings  and  case  studies  to  the  participants;  a 
peer  review  of  the  panel’s  product)  to  ensure 
that  the  resulting  efforts  are  of  greatest  utility  to 
the  US  C4ISR  assessment  community. 

Measures  of  Merit.  To  respond  to  the  needs  of 
new  world  disorder  missions,  efforts  are  needed 
to  formulate  the  upper  levels  of  the  MoM 
hierarchy.  For  example,  in  the  mission  areas  of 
smaller  scale  contingencies  and  operations  other 
than  war,  measures  are  needed  to  characterize 
political-institutional-social  stability.  For 
example,  are  the  bodies  of  government 
functioning  effectively?  Are  educational 
institutions  operative?  Are  children  playing  on 
the  soccer  fields?  Similarly,  in  the  area  of 
infrastructure  assurance,  MoMs  are  needed  to 
reflect  public  confidence  in  key  infrastructures 
(e.g.,  confidence  in  the  finance  and  banking 
sector  as  measured  by  the  value  of  the  Dow 
Jones  Industrial  Average  and  the  value  of  the 
dollar  versus  the  Euro).  These  needs  suggest 
that  MORS  should  convene  a  workshop  on 
expanded  C4ISR  MoMs  for  new  world  disorder 
missions. 

To  satisfy  the  needs  of  the  architectural 
community,  steps  should  be  taken  to  explore 
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options  to  develop  MoMs  that  can  be  used  to 
assess  and  compare  architectural  options.  One 
possible  course  of  action  would  ceill  for  OSD  to 
convene  an  Integrated  Process  Team  (IPT)  of 
stakeholder  organizations  to  address  this  issue. 
MORS  could  provide  the  facilitators  to  conduct 
such  an  event. 

Data.  The  depth  and  breadth  of  the  data 
concerns  are  such  that  a  new  institutional 
mechanism  is  needed  to  deal  with  the  totality  of 
the  problem.  This  mechanism  should  deal  with 
the  full  life  cycle  of  the  data  problem,  to  include 
data  standardization;  data  acquisition; 
verifications,  validation  and  certification; 
transformation  (into  a  form  that  is  useful  to  the 
C4ISR  assessment  community);  and  data  storage 
and  access.  Currently,  there  are  a  number  of 
organizations  that  are  involved  with  significant 
facets  of  the  problem.  These  include  OSD’s 
C4ISR  Decision  Support  Center  (DSC),  PA&E’s 
Joint  Data  System  (JDS),  Defense  Modeling  and 
Simulation  Office  (DMSO)  Modeling  and 
Simulation  Operational  Support  Activities 
(MSOSA),  and  J6’s  Joint  Defense  Information 
Infrastructure  Control  System  (JDIICS)  and 
NETWARS  activities.  OASD(C3I)  should  take 
the  lead  in  forging  these  fragmented  efforts  into 
a  complete,  mutually  reinforcing  solution  to  the 
total  problem. 

Tools  and  their  Application.  It  is  important  that 
C4ISR  analysts  be  aware  that  a  broad  mix  of 
tools  exists  beyond  the  usual  simulation  tools, 
and  it  is  important  that  the  analysts  be 
conversant  about  their  strengths  and  weaknesses. 
In  particular,  where  appropriate,  C4ISR 
assessments  should  take  better  advantage  of 
analyses  based  on  basic  operations  research  and 
physical  principles;  structured  frameworks  for 
analysis;  expert  elicitation  tools  (e.g., 
groupware);  wargames;  analytical  systems 
dynamics  models;  applications  of  the  “new 
sciences”  to  deal  with  complex  adaptive 
systems;  and  multifunctional  tools  (e.g.,  tools 
that  can  support  analyses  and  mission  rehearsal). 
In  all  instances,  assessors  should  assiduously 
pursue  efforts  to  perform  responsible  levels  of 
Verification,  Validation  and  Accreditation 
(W&A)  for  the  tools  selected.  MORS  should 
lead  the  way  in  determining  what  is  a 


“responsible  level  of  VV&A.”  [Note;  the 
SIMVAL  99  workshop  that  MORS  is  hosting  in 
1999  should  shed  light  on  this  issue.] 

Because  each  type  of  tool  has  strengths  and 
weaknesses,  it  is  important  to  select  and 
orchestrate  a  complementary  set  of  tools.  In  the 
past,  this  concept  has  been  limited  to  the  model- 
test-model  paradigm.  However,  it  would  be 
appropriate  to  generalize  that  concept  to 
incorporate  a  broader  set  of  tool  types.  For 
example,  it  might  be  desirable  to  consider  an 
“expert  elicitation-real  world  experience-test- 
model”  paradigm,  that  expands  the  set  of  tools 
that  are  orchestrated. 

Provider-Consumer  Relations.  The  MORS 
Mini-Symposium  on  “Quick  Reaction  Analysis 
Requirements  and  Methodologies  (QRAM)” 
[Ref.  4]  recognized  cuid  addressed  the  issue  of 
provider-consumer  relations.  That  workshop 
recognized  that  there  should  be  a  “contract” 
between  providers  and  consumers,  recognizing 
the  mutual  responsibilities  of  both  sides.  It  is 
recommended  that  the  draft  “contract”  be 
updated  and  refined  to  reflect  the  lessons  learned 
from  this  workshop.  As  a  further  step,  a  COBP 
should  be  developed  (perhaps  in  the  form  of  a 
check  list)  to  help  educate  consumers  about  the 
attributes  of  sound  C4ISR  analysis.  This  “check 
list”  could  be  patterned  after  the  COBP  that  the 
AIAA  Information  and  C2  System  Technical 
Conunittee  is  developing  for  OSD  on  C2 
experimentation. 

3.3.6  Synthesis  Group  Summary.  The  situation 
with  respect  to  the  current  state  of  C4ISR 
analysis  runs  the  gamut  from  bad  news  to  good 
news.  The  bad  news  is  that  the  C4ISR 
assessment  problem  is  getting  substantially  more 
difficult.  We  are  confronted  with  a  host  of  new 
world  disorder  missions  and  issues  for  which  we 
lack  the  key  elements  of  effective  C4ISR 
analyses  (particularly  relevant  data  and  tools). 

In  addition,  these  issues  are  in  a  current  state  of 
flux,  making  it  difficult  to  get  traction  on  the 
problem.  One  of  the  most  dramatic  examples  of 
this  challenge  is  the  infrastructure  assurance 
problem,  where  the  infrastructures  and  the 
threats  to  them  are  changing  dramatically.  More 
bad  news  is  that  currently,  we  generally  do  not 
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do  a  satisfying  job  in  performing  C4ISR 
analyses.  Although  there  are  counter-examples 
to  this  statement,  it  is  recognized  that  most 
studies  of  conventional  conflict  fail  to  live  up  to 
the  NATO  COBP. 

The  good  news  is  that  we  are  beginning  to 
understand  a  great  deal  more  about  the  C4ISR 
assessment  problem.  We  are  starting  to 
recognize  the  importance  of  C4ISR  in  the 
context  of  the  many  missions  that  DoD  must 
perform.  We  are  also  starting  to  recognize  what 
we  don’t  yet  understand  and  the  practices  that 
should  be  followed  to  perform  credible  C4ISR 
analyses.  Finally,  this  workshop  itself  provides 
some  more  good  news.  If  the  recommendations 
of  the  workshop  are  implemented  successfully,  it 
will  provide  the  foundation  for  the  C4ISR 
community  to  work  collaboratively  to  attack  the 
most  critical  of  our  shortfalls. 

4.  INDIVIDUAL  WORKING  GROUP 
SUMMARIES 

The  above  sections  discussed  the  observations, 
findings  and  recommendations  of  the  Synthesis 
Group  aggregated  over  all  of  the  working 
groups.  The  sections  below  provide  similar 
products  for  each  of  the  working  groups.  Each 
section  concludes  with  the  assessment  of  the 
working  group  as  to  the  health  of  C4ISR 
analysis  in  the  mission  or  specialty  area.  The 
discussions  below  represent  only  a  summary  of 
the  key  points  from  the  deliberations  of  the 
working  groups.  More  details  £ire  provided  in 
the  complete  annotated  working  group  briefings 
that  are  found  in  the  appendices. 

4.1  Major  Theater  War  (MTW)  Working 
Group.  The  MTW  working  group  was  co¬ 
chaired  by  Mr.  Charles  Taylor,  C4ISR  Decision 
Support  Center  and  Dr.  Mark  Youngren,  MITRE 
Corporation.  Although  the  MTW  area  is 
certainly  the  most  mature  of  the  mission  areas 
with  respect  to  analysis  attention,  the  working 
group  still  found  serious  deficiencies  in  its 
assessments. 

4.1.1  Analysis  Setup  and  Problem  Definition. 
The  group  felt  that  the  most  meaningful  part  of 
the  analytical  process  is  the  clear  articulation  of 


the  questions  and  working  with  the  decision 
maker  to  ensure  the  question  is  properly 
understood  by  all  and  framed  so  that  an  answer 
can  be  derived.  It  is  essential  to  clearly  define 
the  objectives  and  constraints  and  to  have  good 
scenarios  and  data. 

4.1.2  Models  and  Simulations.  Models  and 
simulations  are  the  workhorse  tools  for  C4ISR 
analysis  in  the  MTW  area,  but  the  legacy  M&S 
tools  are,  for  the  most  part,  models  that  were  not 
built  to  analyze  Joint  problems  and  they  do  not 
model  most  aspects  of  C4ISR  very  well.  For 
example,  it  is  usually  the  case  that  doctrine  is 
hard  wired;  C2  is  scripted;  there  is  limited 
explicit  representation  of  C4ISR  systems; 
limited  ability  to  map  the  contributions  of 
C4ISR  systems  to  combat  outcomes;  fusion  is 
represented  poorly  (if  at  all);  human  factors  are 
not  represented  at  all;  Red  C2  is  not  reactive  to 
Blue  actions;  COMINT  and  HUMINT  are  not 
modeled  well;  and  the  flow  of  information  is 
poorly  represented. 

Other  serious  deficiencies  in  our  existing  M&S 
tools  are  the  modeling  of  non-combat.  Red 
C4ISR,  decision  heuristics  for  Red  mles  of 
engagement,  information  operations,  the  play  of 
coalitions  and  the  representation  of  the  effect  of 
maneuver  on  Red  behavior. 

Current  state  of  the  art.  The  working  group 
raised  concerns  about  the  ability  of  the  various 
simulations  under  development  (e.g.,  JWARS) 
to  significantly  improve  on  the  above 
deficiencies.  They  believe  that  much  research  is 
needed  in  areas  such  as  perception 
development,  fusion,  human  factors,  decision 
making  and  on  reflecting  C4ISR  outcomes  into 
military  effects  like  maneuver.  The  “best  of 
breed”  philosophy  (selecting  the  best  of  the 
existing  algorithms)  of  JWARS  and  other 
simulation  development  efforts  offers  little 
promise  that  those  models  will  meet  the  needs  of 
the  C4ISR  analyst. 
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The  working  group  spent  a  considerable  amount 
of  time  discussing  the  relative  advantages  and 
disadvantages  of  four  analysis  approaches: 

(1)  Single  monolithic  campaign  model  that  is 
responsible  for  representing  everything. 

(2)  Hierarchy  of  models. 

(3)  Federation  of  models. 

(4)  Techniques  other  than  simulation. 

On  balance  the  group  supported  the  federation 
approach  as  having  the  most  promise  for 
analyzing  the  contribution  of  C4ISR  on 
battlefleld  outcomes.  The  federation  approach 
allows  fidelity  where  fidelity  is  needed,  and  it 
allows  the  analytical  team  to  review  the  cause 
and  effect  relationships  as  they  unfold  at  the 
seams  of  the  individual  models.  However,  the 
approach  still  falls  prey  to  such  problems  as 
aggregation/disaggregation,  inconsistent  model 
treatments  and  (W&A). 

The  group  discussed  non-simulation 
methodologies  such  as  spreadsheets,  systems 
dynamics  analytical  models,  wargames, 
exercises,  historical  analyses,  surveys,  man-in- 
the-loop  simulations,  but  it  felt  that  constructive 
simulations  would  continue  to  be  the  main  tool 
for  C4ISR  analysis  in  the  future. 

4.1.3  Measures  of  Merit.  The  problem  is  not 
the  definition  of  C4ISR  MOPs  or  battle  MOEs. 
Rather,  it  is  the  mapping  of  C4ISR  MOPs  into 
battle  MOEs,  the  establishment  of  direct 
linkages  between  the  C4ISR  system 
performance  measures  and  the  criteria  for 
mission  effectiveness.  The  mapping  is  very 
dependent  on  context.  Analysis  requires 
multiple  measures;  no  single  MOP  or  MOE 
satisfies  as  a  global  metric  for  performance  or 
effectiveness  across  all  mission  areas,  nor  is  it  a 
sound  analytical  practice  to  measure  only  a 
single  attribute  as  the  criterion  in  a  study  effort. 

4.1.4  Scenarios.  High  level  scenarios  such  as 
the  Defense  Planning  Guidance  (DPG)  do  not 
specifically  address  the  totality  of  the  variables 
associated  with  setting  up  an  analysis.  This 
requires  the  analyst  to  create  portions  of  the 
scenario  for  a  particular  effort  (e.g.,  specific 
force  laydown,  enemy  behavior,  own  force 
ROE,  etc.).  All  of  these  assumptions  result  in  a 


multiplicity  of  scenarios  eventually  being 
created  from  what  is  believed  by  senior  leaders 
to  be  a  baseline  from  which  all  analysis  efforts 
depart.  Another  serious  problem  with  most 
scenarios  is  the  fact  that  deployment  timelines 
for  C4ISR  assets  are  rarely  represented.  Thus, 
C4ISR  asset  movements  into  theater  are  rarely 
accounted  for  —  tending  to  overstate 
performance.  The  group  also  questioned  if  it 
was  appropriate  in  today’s  environment  to 
continue  to  focus  on  full  blown  MTW  events 
rather  than  the  more  likely  multiple  smaller  scale 
regional  contingencies. 

4.1.5  Experiment  Design.  There  is  a  serious 
problem  with  the  sheer  number  of  variables  to 
be  considered  in  an  analysis  and  the  resulting 
overflow  in  dimensionality.  The  common 
approach  to  deal  with  the  problem  has  usually 
been  to  reduce  (somewhat  arbitrarily)  the 
number  of  variables  for  which  the  analysis  will 
consider.  In  so  doing,  the  analytical  team  mns 
the  risk  of  not  completely  exploring  the  cause 
and  effect  relationships  and  dependencies 
between  multiple  variables.  The  trade-off  is 
obviously  time  and  money.  More  research  is 
needed  as  to  how  one  effectively  manages  the 
overflow  in  dimensionality.  Unfortunately,  most 
analysts  lack  an  appreciation  for  the  statistical 
issues  around  the  experimental  design  and 
interpretation  of  analysis  results. 

4.1.6  Data.  The  group  considered  three  types  of 
data;  system  performance  data,  scenario  data 
and  behavioral  data.  They  judged  the  systems 
data  to  be  the  most  mature;  however  future 
systems  data  are  lacking.  Scenario  data  are 
incomplete  in  its  representation  of  Red  forces 
especially  in  terms  of  Red  C4ISR  systems  and 
Red  behavioral  data.  The  existing  data  don’t 
necessarily  map  into  DPG  scenarios.  Another 
weakness  is  the  absence  of  background  data 
(civilian  traffic,  background  communications 
load,  etc.).  The  behavioral  data  is  the  weakest 
link.  There  are  virtually  no  reliable  data  sources 
for  C2  events  that  require  decision  making  to 
occur  or  data  that  suggests  what  courses  of 
action  may  transpire  from  equivalent  battlefield 
events.  Much  research  is  needed  for  this  area. 
The  group  felt  it  was  necessary  to  have  better 
standardization  and  cataloguing  of  data. 
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4.1.7  Human  Factors  and  Organization.  The 
group  felt  that  this  area  was  the  weakest 
element.  Behavior  models  are  not  developed;  no 
Red  behavior  models  are  available;  data  for 
behavior  and  decision  making  are  not  available; 
human-in-the-loop  approaches  are  not  repeatable 
and  may  not  be  verifiable;  and  we  don’t  know  if 
we  should  model  doctrine,  “real-world,” 
normative  or  optimal  decisions. 

4.1.8  Risk  and  Uncertainty.  Uncertainty  is 
often  modeled  with  probability  distributions. 
However,  we  don’t  understand  the  models, 
much  less  the  distributions,  much  less  the 
distributional  parameters. 

4.1.9  General  Comments.  Some  of  the  most 
critical  issues  surrounding  studies  have  little  to 
do  with  the  traditional  analysis  activities. 

Instead,  they  revolve  completely  around 
exogenous  factors  that  frequently  end  up 
determining  the  results  out  of  hand.  These 
variables  include  the  time  allowed  to  complete 
the  effort,  the  resources  provided,  the  politics  of 
both  the  question  to  be  answered  and  the 
organizations  chosen  to  run  it  (as  well  as  those 
left  out)  and  the  expectations  of  the  decision 
makers  surrounding  the  results.  These  issues  in 
a  strict  scientific  sense  should  have  no  bearing 
on  the  results  and  yet  they  frequently  are  the  first 
order  effects  that  determine  the  answer. 

4.1.10  The  Bottom  Line.  Much  basic  research  is 
still  needed.  The  major  deficiencies  are: 

•  Linkage  between  C4ISR  effects  and 
campaign  outcomes. 

•  Fusion. 

•  Perception  development. 

•  Decision  making. 

•  Basic  combat  itself  (e.g.,  maneuver  effects). 

JWARS  and  other  new  model  efforts  are 
incorporating  the  “best”  current  algorithms  — 
this  approach  will  not  work  with  respect  to 
C4ISR.  We  recommend  developing  a  prioritized 
list  in  the  areas  of  C2,  information  fusion, 
communications,  ISR,  basic  combat  and  C4ISR 
effects  on  combat.  Use  that  prioritized  list  to 


direct  the  research  plan.  This  research  needs  to 
be  implemented  in  new  model  developments, 
but  the  effort  should  not  be  focused  on  a 
particular  tool. 

There  are  fundamental  data  gaps  that  hinder  the 
ability  to  support  C4ISR  analyses.  We 
recommend  that  the  data  requirements  to  support 
C4ISR  analyses  be  identified,  collected  and 
made  available  to  the  analysis  community. 

C4ISR  analysis  will  require  a  range  of  different 
models  (it  will  not  be  limited  to  JWARS).  There 
is  still  a  serious  need  for  additional  development 
of  the  analysis  infrastructure  (to  include  data, 
scenarios,  tools  and  experienced  analysts)  to 
respond  to  current  demands. 

4.1.11  Assessments. 


Assessment  of  the  Health 
of  C4ISR  Analysis  for  MTW 


Element 

Aggregate  Assessment 

Problem  Structuring 

Yellow 

Human  Factors/ 
Organization 

Red 

Scenarios 

Orange 

Measures 

Yellow 

Data 

Red 

Tools 

Red 

Risk  and  Uncertainty 

Red 

Report 

Orange 

4.2  Smaller  Scale  Contingencies  (SSC) 
Working  Group.  In  order  to  differentiate  itself 
clearly  from  OOTW,  the  working  group  decided 
to  focus  its  efforts  on  mission  types  in  which  the 
use  of  military  force  was  likely  (raids,  hostage 
rescue,  opposed  Non-combatant  Evacuation 
Operations  (NEO),  etc.).  The  missions  were 
also  viewed  as  those  for  which  military 
considerations  tend  to  take  precedence  over 
political  considerations.  The  group  recognized 
that  this  is  an  artificial  distinction.  Many  formal 
definitions  of  OOTW  explicitly  include  SSC. 
Moreover,  SSC  intended  as  nearly  pure  military 
operations  often  are  changed  into  more  political 
activities.  The  scenario  played  out  in  Haiti 
(which  went  in  a  matter  of  a  few  minutes  from 
an  invasion  of  a  small  country  to  a  friendly 
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occupation  with  primarily  political  goals) 
provided  a  good  example  of  this  phenomenon. 
Finally,  as  the  group  stressed  in  its  deliberations, 
SSC  tend  to  have  significant  political 
dimensions,  even  when  the  operations 
themselves  are  largely  military  in  character.  The 
working  group  also  concluded  that  analysis  of 
SSC  cannot  be  meaningfully  conducted  on  the 
group  of  missions  as  a  whole.  Their  diversity 
requires  that  tools  and  analytic  efforts  be 
focused  on  meaningful,  coherent  subsets  of  the 
topic.  For  example,  studies  of  raids  will  require 
different  tools  and  data  than  studies  of  hostage 
rescues.  Meaningful  analyses  must  have 
baselines  for  assessment  of  the  contribution  of 
current  and  future  C4ISR  systems,  and  the 
baselines  do  not  exist  for  most  of  these  mission 
types. 

4.2.1  Characterization  of  C4ISR  in  SSC.  The 
dominant  requirement  will  be  for  more  rapid  and 
agile  intelligence,  planning  and  execution 
processes.  This  means  exploring  ways  to 
improve  the  speed  and  responsiveness  of  the 
entire  ISR  process  (from  tasking  through 
dissemination)  as  well  as  to  couple  situation 
awareness,  decision  making  and  execution  more 
closely.  SSC  are  different  from  MTW  in  that 
they  almost  always  involve  considerable 
pressure  for  centralized  control.  This  arises  both 
from  the  legitimate  concerns  of  higher  commcuid 
arising  from  the  potentially  strategic 
implications  of  military  activities  down  to  the 
tactical  level  and  the  more  idiosyncratic  fact  that 
SSC  are  often  "the  only  game  in  town,"  which 
means  that  decision  m^ers  at  all  levels  are  free 
to  invest  time  in  them.  MTW,  on  the  other  hand, 
are  often  large  and  complex  enough  that  senior 
political  and  military  officials  often  are  too  busy 
to  focus  on  tactical  level  decisions  and  actions. 

SSC  may  (but  do  not  always)  require  a  high 
level  of  tactical  security  (particularly  in  the 
planning,  preparation  and  entry  phases)  and 
management  of  highly  visible  consequences. 

The  experience  of  conducting  an  operation  that 
is  a  military  success  yet  fails  in  terms  of  US 
policy  was  all  too  familiar  to  the  working  group 
participants. 


Communication  issues  arose  several  times 
during  the  discussion.  SSC  often  require  new  or 
novel  infrastructures  that  overcome  gaps  arising 
from  military  service  differences,  the  need  to 
include  coalition  partners  or  host  nations,  or  the 
need  to  deal  with  non-military  actors  (inter¬ 
agency  or  NGO).  Moreover,  secure 
communications  may  be  needed  to  achieve 
adequate  operational  security. 

SSC  are  not  necessarily  a  lesser-included  case  of 
MTW.  This  is  crucial  when  data  is  being 
selected,  models  chosen,  parameters  estimated 
or  research  is  being  organized.  Some  of  the 
distinctions  were  mentioned  earlier.  Others 
include  the  fact  that  the  MOFE  and  MOPE  are 
usually  not  under  the  control  of  the  military.  In 
this  sense,  SSC  are  properly  understood  to  be 
OOTW.  Moreover,  the  group  stressed  the  fact 
that  the  intelligence  required  in  many  SSC  is 
more  fine  grained  and  detailed  than  that  required 
for  the  same  levels  of  command  in  MTW. 

SSC  are  often  conducted  under  restraints  that 
would  not  be  present  in  MTW.  Rules  of 
engagement  and  political  restrictions  often  limit 
the  activities  available  for  mission 
accomplishment.  Moreover,  SSC  tend  to  be 
characterized  by  a  very  meaningful  tension 
between  efforts  to  avoid  mission  failure  and 
maximizing  the  likelihood  of  mission  success. 
Examples  of  decisions  to  limit  the  size  and 
armament  involved  in  operations  in  order  to 
avoid  undesirable  political  consequences  were 
offered  by  several  members  of  the  working 
group. 

4.2.2  Assessing  Relative  Worth.  The  method 
for  assessing  the  relative  worth  of  C4ISR 
systems  was  seen  as  readily  available  from 
fundamental  operations  research  techniques. 

This  is  not  to  say  that  the  data  and  tools  needed 
to  conduct  those  analyses  are  available.  The 
group  recognized  the  value  of  a  particular 
C4ISR  innovation  could  be  seen  as  a  function  of 
the  range  of  SSC  missions  to  which  it  applies, 
the  relative  likelihood  of  those  mission  types  and 
their  estimated  impact  on  the  relevant  MOFE 
and  MOPE. 
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4.2,3  Measures  of  Merit.  The  group  readily 
accepted  both  the  set  of  measures  of  merit 
categories  defined  by  MORS  in  the  1980s  and 
the  suggested  change  offered  by  Dr.  Stuart  Starr 
during  his  plenary  address,  of  adding  an  outside 
layer,  Measures  of  Policy  Effectiveness 
(MOPE).  This  addition  leads  to  the  following 
hierarchy  of  measures:  dimensional  parameters, 
MOP,  MOE,  Measures  of  Force  Effectiveness 
(MOFE)  and  MOPE.  Equally  important,  the 
group  also  wanted  to  ensure  that  analyses  of 
C4ISR  in  SSC  "be  informed  by"  metrics  from  all 
the  relevant  layers.  Interoperability  issues  also 
arose  several  times.  Metrics  for  interoperability 
and  for  information  dissemination  remains 
important  in  assessing  future  C4ISR  systems. 

The  quality  of  the  information  available  was 
seen  as  the  single  most  important  metric  for 
SSC.  This  was  defined  to  include  its 
responsiveness  to  warfighters,  including  the 
Commander's  Critical  Information  Requirements 
(CCIR),  but  also  the  needs  of  key  staff  members 
and  the  units  executing  the  operation. 
Responsiveness  was  seen  as  including  much 
more  than  mere  availability. 

What  is  meant  by  high  quality  information? 
"Perfect"  information  in  a  military  situation  is 
seen  as  complete  (for  the  command's  needs), 
current,  correct,  precise  enough  for  the 
associated  military  purposes  (which  will  vary 
from  targeting,  which  is  very  precise,  to  general 
maneuver  planning,  which  is  often  much  less 
precise)  and  consistent.  Consistent  here  means 
both  internally  consistent  within  the  data, 
information  and  knowledge  of  the  C4ISR  system 
and  consistent  across  command  nodes. 

Measures  of  merit  that  focus  on  rapid  and  robust 
decision  making,  planning  and  execution  will  be 
needed.  This  means  we  need  valid  and  reliable 
indicators  of  the  speed  of  the  C4ISR  processes, 
the  variety  of  futures  and  alternative  courses  of 
action  they  can  and  do  consider,  and  the  quality 
of  the  decisions  they  make.  Indicators  of 
decision  quality  essentially  must  be  MOFE  and 
MOPE,  though  MOE  for  C4ISR,  MOP  for 
C4ISR  systems  and  the  dimensional  parameters 
that  drive  them  may  also  be  needed  for 
diagnostic  purposes  in  any  given  analysis. 


4.2.4  Tools.  This  area  was  perhaps  the  least 
mature  of  the  areas  discussed  in  the  working 
group.  The  most  important  tool  requirement 
was  for  credible  support  to  Course  Of  Action 
(COA)  analysis.  This  means  tools  that  generate 
projections  of  alternative  futures  that  are 
credible  to  operators  as  well  as  the  capacity  for 
sensitivity  or  "what  if  analyses.  These  tools 
must  also  be  fast  enough  to  support  mission 
planning  and  have  an  analytic  mode  that  permits 
large  numbers  of  runs  in  a  short  period  of  time. 
Moreover,  these  tools  must  be  designed  to  pull 
from  operational  databases,  not  just  notional  or 
analytic  ones. 

Perhaps  the  most  novel  idea  arising  for  new 
tools  was  a  call  for  a  virtual  C4ISR  rehearsal 
capability  that  would  be  designed  both  as  a  tool 
for  analysts  and  for  operators.  Operators  would 
use  it  for  training  and  mission  rehearsal. 

Analysts  could  use  it  for  improving  their 
understanding  of  missions,  generating 
hypotheses  about  C4ISR  improvements, 
examining  man  in  the  loop  elements  and  testing 
hypotheses. 

The  group  heard  a  similar  plea  for  more 
effective  ways  to  both  link  and  differentiate  real 
and  simulated  data,  particularly  in  real 
headquarters  participating  in  virtu2il  exercises 
and  training. 

In  general,  tools  are  needed  to  assess  the  degree 
to  which  C4ISR  systems  are  linked  across 
functions,  echelons  and  time.  New  information 
systems  are  linking  operators  more  and  more 
closely,  but  no  tools  to  measure  these  changes  or 
their  impacts  are  currently  available.  Similarly, 
models  that  capture  the  effects  of  "soft  factors" 
such  as  differences  in  language,  culture  or 
training  are  not  available  but  would  be  valuable 
in  an  era  that  stresses  "jointness"  and  coalition 
operations. 

Analysts,  particularly  those  supporting 
operational  organizations,  also  felt  a  need  to 
capture  and  replay  information  (data,  voice 
traffic,  imagery,  etc.)  for  current  and  future 
operations.  This  would  greatly  facilitate  after¬ 
action  analyses  from  both  operations  and 
exercises. 
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4.2.5  Analysis  of  Four  SSC  Mission  Areas. 

The  group  focused  in  detail  on  four  different 
types  of  SSC  mission  areas  to  link  future  C4ISR 
needs,  the  analytical  tools  required  to  support 
their  development,  the  most  crucial  measures  of 
merit,  and  implied  key  metrics.  The  four 
mission  areas  were: 

1.  Raid. 

2.  Urban  Warfare. 

3.  Counter  Terrorism/Operational  Disruption. 

4.  Opposed/Potentially  Opposed  NEO. 

The  discussions  of  measures  was  focused  on 
MOPE  and  MOPE. 

4.2.6  The  Bottom  Line.  The  current  and  future 
C4ISR  needs  for  SSC  must  be  addressed  within 
the  context  of  each  specific  mission  category,  as 
opposed  to  addressing  a  broad  range  of  similar 
missions.  By  heeding  this,  proper  analysis  of 
each  mission  area  may  be  secured.  C4ISR  data 
within  SSC  will  need  to  develop  a  higher  level 
of  tactical  detail  and  control  then  other  arenas  of 
warfare  (i.e.  peacekeeping,  peace  enforcement, 
major  theatre  war.)  This  characteristic, 
developed  from  tight  operational  constraints, 
will  aid  commanders'  tactical  planning,  control 
and  security.  Investment  in  such  C4ISR 
information  systems  such  as  virtual  rehearsal 
programs,  course  of  action  analytical  tools,  and 
relational  databases  is  recommended  to  increase 
military  force  flexibility  and  performance. 

4.2. 7  Assessments. 


Assessment  of  the  Health  of 
C4ISR  Analysis  for  SSC 


Element 

Aggregate  Assessment 

Problem  Structuring 

Orange 

Human  Factors/ 
Organization 

Orange 

Scenarios 

Yellow 

Measures 

Orange 

Data 

Red 

Tools 

Orange 

Risk  and  Uncertainty 

Orange 

Report 

Red 

4.3  Operations  Other  than  War.  As  was  done 
by  the  SSC  working  group,  the  OOTW  Working 
Group  began  their  deliberations  by  scoping  the 
definition  of  OOTW  to  exclude  any  mission  in 
which  combat  might  be  anticipated.  The 
working  group  believed  the  fact  that  OOTWs  are 
going  on  near-continuously  offered  the 
opportunity  to  focus  analysis  efforts  on  real 
operations  —  observe  and  analyze  the 
operations  as  they  occur.  This  reduces  the  need 
to  build  models  and  simulations. 

4.3.1  Problem  Decomposition.  The  OOTW 
group  decomposed  the  problem.  For  a  set  of 
OOTW  missions,  they  derived  a  set  of  typical 
functions  within  the  missions  and  identified 
typical  mission  scenarios  for  analysis.  For  the 
scenarios,  they  considered  how  C4ISR  means 
were  employed  in  the  mission.  By  crossing  the 
C4ISR  means  with  the  mission  functions,  they 
identified  the  most  interesting  C4ISR  evaluation 
issues.  Finally,  these  led  to  a  categorization  of 
typical  measures  of  merit  and  the  appropriate 
tools  to  assess  them. 

The  decomposition  effort  partitioned  the 
problem  into  12  major  functions  associated  with 
OOTW  missions:  security  operations,  police 
operations,  refugee  operations,  resettlement, 
compliance,  inspection,  surveillance,  PSYOPS 
(Perception),  civil  affairs,  sustenance,  medical 
operations  and  governance.  They  then  mapped 
the  functions  to  the  following  subset  of  OOTW 
missions: 

•  Peace  Keeping  (PK) 

•  Peace  Enforcement  (PE) 

•  Peace  building 

•  Humanitarian  Assistance/Disaster  Relief 
(HA/DR) 

•  Civil  support 

•  Nation  assistance 

•  Counter  drag 

•  Counterinsurgency  support 

They  then  focused  on  two  different  missions: 
PK/PE  and  HA/DR  and  characterized  each. 

This  activity  demonstrated  clearly  that  the  issues 
for  the  different  missions  were  significantly 
different.  For  example,  PK/PE  focuses  on 


16 


preventing  conflict,  restoring  governance  and 
stabilizing  the  situation,  whereas,  HA/DR 
focuses  on  stopping  the  suffering  and  dying. 
Similarly,  there  are  different  types  of 
participants  (coalition  forces,  civilian 
authorities,  non-government  organizations,  etc.), 
levels  of  force,  different  data  required  and 
different  measures. 

4.3.2  Tools.  The  working  group  decided  that 
many  different  types  of  tools  would  be  needed  to 
address  the  spectrum  of  OOTW  missions.  These 
include:  Expert  elicitation,  polling,  statistical 
analysis,  data  mining,  wargaming,  collaborative 
decision  making,  network  analysis,  functional 
capability  models,  learning  from  real-world 
activities,  historical  analyses,  analytic 
frameworks  and  predictive  capability  models. 

Of  course  the  group  felt  that  the  usual 
simulations  would  also  be  useful,  but  they  would 
be  less  important  for  supporting  analyses  of 
C4ISR  in  the  OOTW  mission  area  than  in  those 
mission  areas  that  focused  on  military 
operations. 

While  an  end-to-end  modeling  approach  in 
OOTW  is  well  beyond  the  present  state  of  the 
art,  there  is  a  clear  need  for  tools  and  models 
that  will  do  part  of  the  job  within  the  context  of 
an  analytic  framework.  For  example,  tools  for 
HA/DRs  that  help  to  predict  damage  and 
casualties  and  generate  requirements  for  medical 
supplies  and  personnel  are  needed.  Within 
peace  operations,  models  for  assessing  the 
performance  of  different  ISR  capabilities  or 
communications  networks  would  be  helpful. 
Tradeoffs  frequently  will  be  based  on  expert 
judgment  and  will  be  qualitative  rather  than 
quantitative.  In  this  area,  qualitative  trade-off 
judgments  may  be  better  than  judgments  based 
on  trying  to  force  the  numbers  to  fit  the  problem. 

Trades  of  C4ISR  with  forces  for  OOTW  require 
better  definition  of  missions  and  requirements, 
more  organized  exploitation  of  past  lessons 
learned,  databases  of  operational  data, 
frameworks  for  analysis  and  appropriate 
supporting  tools. 

4.3.3  Measures.  OOTW  mission  measures  are 
known  but  may  vary  widely  by  mission.  The 


measures  are  standard  for  many  trades  within 
C4ISR  (e.g.,  command  and  control  vs 
communications),  but  are  very  difficult  for 
making  tradeoffs  between  elements  of  C4ISR 
and  forces.  Certainly,  by  the  nature  of  the 
mission,  the  measures  needed  for  the  latter 
tradeoff  were  not  the  usual  attrition  measures. 

4.3.4  Recommendations.  Establish  an  effort  to 
advance  our  knowledge  of  OOTW  and  C4ISR  in 
OOTW  leading  to  best  practices  and  analytic 
frameworks.  Continue  efforts  to  identify 
analytic  tool  requirements  to  assess  C4ISR 
support  to  OOTW.  Continue  development  of 
methods  to  better  integrate  OOTW  analysis  into 
DoD  planning  and  programming.  Establish  a 
common  syntax  and  semantics  to  facilitate  a 
comparison  between  study  results.  Develop  a 
deliberate  approach  to  data  collection  (identify 
and  prioritize  needs,  establish  a  mechanism  for 
collecting  data,  establish  repositories  and 
include  non-DoD  sources. 

4.3.5  Assessments. 


Assessment  of  the  Health  of  C4ISR 
Analysis  for  OOTW 


Element 

Aggregate  Assessment 

Problem  Structuring 

Orange 

Human  Factors/ 
Organization 

Red 

Scenarios 

Yellow 

Measures 

Orange 

Data 

Red 

Tools 

Orange 

Risk  and  Uncertainty 

Red 

Report 

Orange 

4.4  Peacetime  Engagement  Working  Group. 
The  Peacetime  Engagement  (PE)  working  group 
immediately  came  face  to  face  with  a  definition 
problem.  There  is  no  current  Joint  Staff 
definition  of  PE.  The  group  discussed  the 
relationship  between  PE  and  overseas  presence, 
and  decided  on  the  following  definition  which 
characterizes  PE  as  deterrent  in  nature. 

Definition:  Peacetime  Engagement  is  the 
application  of  resources  (including  forces)  in 
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peacetime  to  make  shooting  less  likely. 
Successful  PE  would  promote  other  goals: 

•  Stability  of  commerce. 

•  Deterrence. 

•  Separation  of  potential  belligerents. 

•  Exercises  with  (potential)  allies. 

Information  superiority  was  judged  to  be  the 
principal  element  of  C4ISR  implementation  for 
the  future.  Successful  management  of 
information  will  depend  upon  development  and 
establishment  of  joint  doctrine,  including 
processes  and  organizations. 

4.4.2  Measures  of  Merit.  The  working  group 
devoted  much  of  its  time  discussing  measures  of 
merit.  They  partitioned  the  measures  as  falling 
into  five  categories: 

1.  Level  of  effort. 

2.  Task  performance. 

3.  Mission  progress. 

4.  Political  goals. 

5.  Costs. 

They  discussed  a  variety  of  measures  within 
each  category,  some  of  which  were  standard 
measures  like  number  of  reconnaissance  flights 
flown  and  ISR  reports  delivered.  However, 
many  were  measures  unlike  those  used  for 
combat  operations.  For  example,  the  price  of  oil 
and  the  number  of  civilian  truck  deliveries.  To  a 
large  extent  the  measure  of  success  for  a  PE 
mission  is  the  extent  to  which  daily  activities  are 
routine. 

4.4.3  Insights.  The  group  discussions  generated 
several  useful  insights  about  C4ISR  analysis  in 
the  PE  mission  area.  Some  of  the  insights  were: 

•  Information  operations  highest  utility  will  be 
in  PE. 

•  Intelligence  Preparation  of  the  Battlefield 
(IPB)  is  conducted  during  PE. 

•  PE  is  not  a  lesser  included  case  of  warfare. 

•  Coalitions  are  important;  there  will  be 
greater  numbers  of  potential  partners  in  PE 
than  in  specific  MTWs. 


•  Interoperability  of  C3I  systems  will  be 
critical  for  PE  because  of  the  many  potential 
partners. 

•  PE  requires  drawing  in  political 
constraints/considerations  to  a  degree 
greater  than  for  other  mission  areas. 

4.4.4  Data.  Like  the  other  mission  areas,  the 
working  group  judged  the  availability  of  data  to 
be  a  serious  problem.  The  significant  role  of 
political  constraints  generates  peculiar  data 
requirements.  Furthermore,  the  imcertainty 
about  the  probabilities  of  specific  threats  creates 
significant  data  collection  and  distribution 
needs. 

4.4.5  Assessments.  Because  PE  is  not  very  well 
defined,  the  state  of  the  practice  for  analysis 
related  to  any  aspect  of  PE  has  many  gaps. 


Assessment  of  the  Health  of  C4ISR 
Analysis  for  PE 


Element 

Aggregate  Assessment 

Problem  Structuring 

Yellow 

Human  Factors/ 
Organization 

Yellow 

Scenarios 

Green/Y  ellow 

Measures 

Orange 

Data 

Orange 

Tools 

Orange 

Risk  and  Uncertainty 

Red 

Report 

Orange 

4.5  Infrastructure  Assurance.  Since 
Infrastructure  Assurance  (lA)  is  a  relatively  new 
and  evolving  mission  area  addressing  a  new  kind 
of  warfare  unlike  what  we  have  experienced  in 
conventional  warfare,  the  working  group  spent 
its  initial  discussions  defining  the  mission  area 
and  being  educated  about  recent  Presidential  and 
DoD  activities  related  to  the  area.  Following  is 
the  official  definition  of  Critical  Infrastructure 
Protection  (CIP)  (CIP  and  lA  were  considered  to 
be  synonymous)  provided  in  the  Presidential 
Decision  Directive  (PDD)  63,  signed  in  May 
1998. 
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4.5.1  Definition.  Infrastructure  assurance  is  the 
planning  to  improve  the  readiness,  reliability  and 
continuity  of  infrastructures  such  that  they  are: 

•  Less  vulnerable  to  disruptions  or  attack. 

•  Harmed  to  a  lesser  degree  in  event  of 
dismption  or  attack. 

•  Can  be  readily  reconstituted  to  re-establish 
vital  capabilities. 

The  DoD  mission  areas  for  lA  are  divided  into 
1 1  infrastructure  sectors  (3  of  which  comprise 
the  C4ISR  mission  area):  defense  information 
infrastmcture,  command,  control  and 
communications,  intelligence,  surveillance  and 
reconnaissance,  public  works,  financial  services, 
transportation,  health  affairs,  emergency 
services,  personnel,  space  and  logistics. 

The  sectors  cannot  be  considered  in  isolation; 
each  is  dependent  on  the  others  to  achieve 
mission  objectives.  Since  these  services  are 
routinely  provided  by  commercial  assets  outside 
the  government  sector,  the  focus  of  analysis 
must  broaden  to  include  law  enforcement.  Other 
Government  Agencies  (OGA)  and  the  private 
sector.  The  dependencies  among  the  sectors 
must  be  well  understood  for  analysis. 

4.5.2  DoD  Approach  to  Infrastructure 
Assurance.  The  draft  DoD  CIP  Plan  states  that 
the  approach  to  lA  is  to: 

•  Identify  relevant  characteristics  of  each 
infrastructure  that  is  critical  to  military 
mission  success. 

•  Analyze  military  plans  to  identify  critical 
infrastructure  assets. 

•  Assess  vulnerabilities  of  critical  DoD  assets. 

•  Redirect  resources  to  reduce  prioritized 
vulnerabilities. 

•  Collaborate  with  other  government  agencies 
and  the  private  sector  to  reduce 
vulnerabilities  when  services  are  provided 
from  outside  the  DoD  fence. 

The  above  steps  are  conducted  in  the  context  of 
six  lifecycle  phases  that  occur  before,  during, 
and  after  events  which  may  compromise  or 


disrupt  the  services  or  mission  accomplishment 
of  a  critical  infrastmcture; 

•  Analysis  and  assessment  (preventive). 

•  Remediation  (preventive). 

•  Indications  and  warning  (before  an  event). 

•  Mitigation  (before  and  during  an  event). 

•  Response  (during  an  event). 

•  Reconstitution  (after  an  event). 

4.5.3  Data.  Operational  architecture  data  on 
organizational  missions,  functions,  stmctures, 
systems  and  their  relationships  to  include 
information  flows  and  dependencies  are  needed. 
This  presents  unique  problems  because  much  of 
the  needed  data  must  come  from  the  private 
sector  and  are  proprietary.  Much  inter-agency 
information  sharing  is  needed  to  satisfy 
requirements.  The  MDITS  database  for  counter¬ 
terrorism  is  one  example  of  a  database  that  must 
be  shared  with  the  lA  community. 

4.5.4  Analysis  Tools.  Current  DoD  analysis 
tools  generally  lack  the  capability  to  facilitate 
detailed  LA  analysis  because  C4ISR 
requirements  generally  have  not  addressed  the 
need  to  support  lA.  lA  tools  have  been 
developed  in  response  to  lA  community 
requirements  and  are  effective  for  analyzing 
commercial  infrastmcture  and  the  military 
missions  that  they  support.  Integrated  and 
interoperable  tools  or  comprehensive  models  for 
analyzing  defense  infrastmctures  are  non¬ 
existent.  Likewise  tools  to  conduct 
interdependency  analyses  are  non-existent. 
Some  of  the  tools  that  ate  used  for  analysis 
supporting  lA  are  the  Joint  C4ISR  Architecture 
Planing/Analysis  System  (JCAPS)  and  the  NCS 
for  communications.  For  cyber  infrastmcture, 
the  NIPC  and  various  Certs  are  C4ISR 
components  that  support  analysis  and 
assessments.  Wargaming  is  frequently  used  to 
better  understand  the  problems  and  issues. 
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4.5.5  Measures  of  Merit.  The  working  group 
suggested  a  hierarchy  of  measures  of  merit  for 
lA  and  its  dependence  on  C4ISR  as  a  basis  for 
mission  success  and  national  security.  The 
hierarchy  (from  top  to  bottom)  is: 

•  National  security. 

•  Missions  (DoD  and  other). 

•  lA  functions. 

•  C4ISR  systems  performance. 

4.5.6  Issues.  Some  of  the  major  issues 
identified  by  the  working  group  are: 

•  Getting  governmental  entities  within  DoD 
and  other  Federal,  state  and  local  agencies  to 
share  information  (which  is  sometimes 
proprietary),  to  share  costs,  and  to  have  a 
common  understanding  of  lA. 

•  Fidelity  of  data. 

•  Getting  the  commercial  sector  to  provide 
proprietary  data. 

•  Developing  a  central  data  repository. 

•  Security  and  privacy  issues. 

•  Legal  issues. 

•  International  cooperation. 

4.5.7 Recommendations.  The  group  made  two 
recommendations: 

1 .  MORS  continue  to  emphasize  lA  as  an 
essential  element  of  C4ISR  (and  other) 
analyses  and  wargaming  and  simulation 
exercises. 

2.  MORS  and  the  lA  community  begin  to 
develop  mutual  understanding  of  the 
requirements  of  lA  analysis  requirement  in 
order  to  develop  integrated  or 
complementary  tools. 


4.5.8  Assessments. 


Assessment  of  the  Health  of  C4ISR  Analysis 
for  Infrastructure  Assurance 


Element 

Aggregate  Assessment 

Problem  Structuring 

Yellow 

Human  Factors/ 
Organization 

Red 

Scenarios 

Red 

Measures 

Red 

Data 

Red 

Tools 

Red 

Risk  and  Uncertainty 

Red 

Report 

Red 

4.6  Information  Architectures.  The 
Architectures  working  group  was  included  as 
part  of  the  workshop  because  of  the  increasing 
emphasis  and  importance  of  operational  and 
systems  architectures  for  information  systems  to 
the  C4ISR  analyst.  Analytical  paradigms  have 
traditionally  focused  on  evaluating  the 
effectiveness  of  a  platform  or  system. 
Architectures  provide  a  structured  approach  for 
describing  and  understanding  a  system-of- 
systems  associated  with  a  specified  domain, 
such  as  C4ISR.  In  so  doing,  this  allows  that 
domain  to  be  considered  as  if  it  were  an  entity. 
Using  the  architecture  construct  enables  our 
analytical  focus  to  evolve  from  being  platform¬ 
centric  to  being  architecture-centric. 

Architectures  are  still  an  evolving  discipline 
which  demands  a  combination  of  both  art  and 
science.  They  help  us  understand  the 
relationships  and  dependencies  among  tasks, 
operational  elements  accomplishing  those  tasks 
and  the  information  flow  that  needs  to  occur 
among  the  operational  elements  to  accomplish 
the  tasks  appropriately.  They  enable  us  to 
describe  the  systems  and  communications  that 
support  the  information  flow  across  nodes  and, 
in  so  doing,  enable  the  understanding  of  how 
information  technology  supports  work.  The 
architecture  descriptions  form  the  basic  data  set 
of  activities,  operational  elements,  information 
flows,  nodes,  systems  and  communications  that 
support  a  wide  variety  of  C4ISR  issue  analyses. 
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An  architecture  should  not  be  viewed  as  an  end 
in  itself,  but  should  be  a  means  to  an  end. 
Operational  architectures  can  capture 
operational  requirements  in  a  fashion  that  allows 
consistent  traceability  to  design  specifications. 

4.6.1  Keys  to  Realizing  the  Potential  of 
Architectures.  The  group  discussed  several 
things  needed  to  realize  the  potential  of 
architectures.  The  keys  are  sununarized  below: 

•  Must  be  able  to  access,  manipulate  and  reuse 
data  across  a  wide  range  of  stakeholders  to 
include  developers,  users  and  acquirers. 

•  Must  be  able  to  integrate  human  factors  into 
the  information  architectures.  People  are  an 
integral  part  of  any  information  architecture. 

•  Need  to  represent  the  threat  more  overtly  to 
include  analysis  of  adversary  as  well  as 
friendly. 

•  Better  tools  are  needed  (better  visualization 
tools,  more  flexible  tools,  more  usable  by 
the  operational  subject  matter  experts). 

•  Must  be  able  to  better  articulate  the 
operational  impacts  of  architectural  analysis 
to  warfighters  and  to  decision  makers. 

4.6.2  Measures  of  Merit.  Measures  are  needed 
for  information  architectures  to  quantify  the 
relationship  between  measures  of  performance 
and  operational  measures  of  effectiveness. 
Measures  are  also  needed  to  evaluate  an 
architecture.  Three  aspects  that  could  be 
measured  are: 

1 .  How  well  the  architecture  description  has 
been  developed. 

2.  To  what  extent  is  the  architecture 
description  being  used. 

3.  The  value  of  the  architecture  in  achieving 
C4ISR  goals  and  objectives. 

4.6.3  Assessments.  Because  information 
architectures  do  not  represent  a  mission  area,  the 
standard  COBP  for  evaluating  the  health  of 
C4ISR  Analysis  is  not  as  relevant  for 
architectures  as  it  is  for  a  mission  area. 
Nevertheless,  the  architectures  working  group 
provided  an  assessment  (with  perhaps  some 


differences  in  the  interpretations  of  the 
elements). 


Assessment  of  the  Health  of  C4ISR  Analysis  for 
Information  Architectures 


Element 

Aggregate  Assessment 

Problem  Structuring 

Green 

Human  Factors/ 
Organization 

Yellow 

Scenarios 

Yellow 

Measures 

Red 

Data 

Yellow/Orange 

Tools 

Yellow 

Risk  and  Uncertainty 

Yellow 

Report 

Red/Orange 

4.7 Analytical  Techniques  and  Tools.  Even 
though  each  of  the  other  working  groups 
addressed  the  issue  of  tools  to  support  C4ISR 
analysis  within  a  specific  mission  area,  MORS 
had  a  separate  working  group  focusing 
specifically  on  analytical  techniques  and  tools 
for  C4ISR  analysis.  The  group  discussed  a 
variety  of  methodologies  to  include  simulations 
such  as  JWARS,  network  analysis  models  such 
as  NETWARS,  optimization  tools,  systems 
dynamics  models  and  Complex  Adaptive 
Systems  (CAS).  However,  the  focus  was  on  the 
workhorse  simulation  tools. 

4.7.1  Taxonomy  of  Decision  Issues.  In  order  to 
structure  the  assessments  and  to  facilitate 
communication,  the  working  group  developed  a 
taxonomy  of  decision  issues.  The  taxonomy 
partitioned  the  C4ISR  analysis  problem  along 
three  dimensions  according  to  the  category  of 
decision  issue  (force  structure,  cognitive 
behavior,  CONORS,  architecture, 
vulnerabilities,  trade  space),  the  decision  maker 
(acquisition,  operational,  POM/budget)  and 
scale  of  conflict  (MTW,  SSC,  OOTW).  Then, 
within  each  element  of  the  cube  the  working 
group  developed  a  qualitative  assessment  of  the 
ability  of  existing  tools  and  techniques  to 
support  C4ISR  analysis.  This  qualitative 
assessment  considered  tools  and  techniques  in 
aggregate  (simulations,  optimization,  CAS, 
systems  dynamics,  game  theory,  etc.)  and 
considered  a  variety  of  evaluation  criteria 
(scenario,  data,  human  behavior,  maturity. 
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measures,  etc.).  For  many  of  the  cells 
(combinations  of  decision  issue  by  decision 
maker  by  mission  area,  comments  were  provided 
to  explain  the  reasons  for  the  assessment.  The 
number  of  combinations  are  too  voluminous  to 
be  presented  in  this  summary;  however  we  will 
provide  some  general  observations  based  on  the 
evaluations.  See  the  briefing  slides  from  the 
Analytical  Techniques  and  Tools  Working 
Group  in  the  Appendix  for  the  all  of  the 
evaluations  provided. 

All  of  the  assessments  for  the  operational  and 
acquisition  decision  makers  were  either  yellow 
or  red  (no  green).  The  situation  was  judged  to 
be  somewhat  better  for  the  POM  decision  maker 
with  a  sprinkling  of  green  assessments  for  the 
CONOPS  decision  category  and  for  force 
structure  (the  area  where  we  have  the  most 
experience)  under  the  MTW  mission  area.  The 
most  frequent  explanation  as  to  why  an 
evaluation  was  rated  yellow  or  red  was  a 
problem  with  data. 

4.7.2  The  Sky  is  Not  Falling.  Even  though  the 
evaluations  of  the  existing  analytical  techniques 
and  tools  painted  a  bleak  picture  as  to  providing 
support  to  the  C4ISR  analysis  community,  the 
group  emphasized  that  the  story  is  not  as  bad  as 
it  may  sound.  There  are  some  promising  new 
analytical  techniques;  there  are  some  promising 
new  simulation  development  efforts;  and  there  is 
a  good  core  of  analytical  talent. 

4.7.3  Recommendations.  The  working  group 
was  optimistic  that  the  problems  could  be  fixed 
with  additional  attention.  Specific 
recommendations  were  to: 

•  Continue  to  fund  research  and  development 
in  the  applications  of  current  methods  and 
new  tools. 


•  Refine  j  oint  analysi  s  tools. 

•  Train  analysts  to  do  analysis  of  joint  issues. 

•  Emphasize  more  effective  ways  to  explore 
the  scenario  space. 

5.  CLOSING  COMMENTS 

This  workshop  purposely  sought  out  individuals 
who  were  not  among  the  “usual  MORS 
workshop  attendees”  for  the  express  purpose  of 
seeking  out  fresh  new  ideas  about  C4ISR 
analysis.  To  a  large  extent  the  memberships 
(and  the  chairs)  of  the  Information  Assurance 
and  the  Information  Architectures  Working 
Groups  were  comprised  of  people  who  were  not 
regular  MORS  workshop  attendees.  There  was 
widespread  agreement  that  this  expanded 
membership  contributed  greatly  to  the  success  of 
the  workshop.  Thanks  should  go  especially  to 
the  chairs  of  those  working  groups:  Mr.  Thomas 
Bozek  and  Mr.  Frank  Ruggeri  for  Information 
Assurance  and  Ms.  Patsy  McGrady  for 
Information  Architectures.  MORS  should 
continue  the  practice  of  seeking  out  "new  blood" 
for  workshops. 

Spectrum  of  Analysis  Techniques.  Analysts 
will  always  have  to  rely  on  clever,  innovative 
use  of  the  entire  spectrum  of  evaluation 
techniques.  Table  5  depicts  the  most  common 
techniques  and  characterizes  each  according  to 
resources,  lead  time,  breadth  of  application, 
replicability  and  credibility.  It  shows  seven 
classes  of  techniques  ranging  from  expert 
elicitation  to  real  crises  and  combat.  A  mix  of 
tools  will  generally  be  required  to  compensate 
for  the  shortfalls  of  individual  tools  and  to 
adequately  represent  Blue  and  Red  processes. 
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Table  5.  Spectrum  of  Evaluation  Techniques. 


Technique 

Resources 

Lead  Time  to 

Create  and  Apply 

Breadth  of 
Application 

Replicability 

Credibility 

Expert 

Elicitation 

Lowest 

Days-Weeks/  Days 

Very  Broad 

Limited 

Variable 

Low 

Limited 

Fair 

Analytical 

Models 

Low 

Weeks/Days 

Broad 

Fully 

Fair 

Constructive 

Simulation 

Low- 

Moderate 

Months/Days 

Broad 

Fully 

Moderate 

Virtual 

Simulation 

High 

Years/ 

Months 

Moderate 

Potential  for 

Good 

Live 

Simulation 

High 

Years/Weeks 

Limited 

Little 

Generally  Good 

Real  Crises/ 
Combat 

N/A 

N/A 

Quite  Limited 

None 

Excellent 

A  mix  of  tools  will  also  be  necessary  to  pursue 
an  iterative  approach  to  C4ISR  analysis.  The  set 
of  techniques  appropriate  for  a  broad  and 
shallow  initial  cut  at  the  problem  will  be 
different  from  the  techniques  appropriate  for 
subsequent  narrower  and  deeper  cuts  at  the 
problem.  In  the  early  stages,  the  techniques  will 
tend  to  be  those  at  the  lower  part  of  the  spectrum 
(expert  elicitation,  wargames,  analytical 
models).  Quick  and  inexpensive  techniques  will 
be  needed  to  efficiently  explore  the  scenario 
space,  to  determine  what  are  the  critical 
parameters  and  to  design  the  analyses  that  will 
follow  in  the  later  stages.  In  the  later  stages,  the 
techniques  will  tend  to  be  more  resource 
intensive  constructive,  virtual  or  live 
simulations. 

5.2  Comments  on  the  Self-Assessments.  The 
COBP  template  proved  to  be  very  useful  for 
focusing  discussions  during  the  workshop.  The 
assessments  of  the  working  groups  about  the 
health  of  the  state-of-practice  for  C4ISR  analysis 
were  not  encouraging,  but  I  suspect  that  we 
tended  to  be  overly  pessimistic  in  our 
assessments.  Yes,  many  improvements  are 
needed  —  particularly  with  respect  to  human 
factors,  scenarios,  data  and  analysis  tools. 
However,  there  was  little  mention  during  the 
workshop  of  perhaps  the  most  important  element 
in  the  conduct  of  good  C4ISR  analysis  —  the 
analyst. 


5.3  The  Importance  of  Good  Analysts.  We 
need  to  keep  in  perspective  that  even  good  tools 
and  data  and  scenarios  do  not  guarantee  good 
analyses.  Instead,  good  analyses  result  from: 

•  The  ability  to  parse  a  problem  down  to  its 
essence. 

•  The  ability  to  synthesize  the  diverse  parts  of 
a  problem. 

•  Domain  expertise. 

•  Availability  of  analysis  tools  (including  data 
and  scenarios). 

•  Knowledge  of  the  tools  and  the  craft  of 
analysis. 

•  Experience  on  the  job. 

•  Knowledge  of  military  operations. 

•  Good  communications  between  the  analyst 
and  the  consumer. 

Most  of  the  above  requirements  for  good 
analyses  point  to  the  need  for  good,  experienced 
analysts.  Analysts  knowledgeable  about  their 
tools  can  frequently  figure  out  ways  to  make 
good  use  of  the  available  tools.  Good  analysts 
can  make  innovative  uses  of  tools  to  address 
questions  that  the  tools  were  never  designed  to 
address.  If  good  tools  are  placed  in  the  hands  of 
good  analysts,  the  combination  is  unbeatable. 
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Gary  Federici,  CNA,  stated  in  his  response  to 
my  inquiries  about  tool  to  support  C4ISR 
analysis  a  few  years  ago: 

I  have  an  uneasy  feeling  that  many  of 
those  who  are  involved  in  all  this  are 
not  clearly  aware  of  some  crucial 
truths  about  modeling.  There  is  a 
widespread  perception,  I  believe,  that 
a  model  is  a  black  box  which  accepts 
assumptions  on  the  input  side  and 
reports  outcomes  as  output. ...  What 
relatively  few  people  reflect  on,  from 
lack  of  direct  experience  in  or  study  of 
campaign  analysis,  is  that  within  the 
black  box  sit  a  group  of  analysts  who 
pull  its  levers  and  turn  its  dials  and 
have  a  very  major  influence  on  its 
outcomes. 

It  is  absolutely  essential  that  DoD  pay  as  much 
(or  more)  attention  to  developing  and  retaining 
good  analysts.  In  the  words  of  Dr.  Stuart  Starr, 
DoD  needs  to  develop  an  “intellectual  reservoir” 
upon  which  to  base  future  C4ISR  analysis 
activities.  Such  a  reservoir  will  require  a 
continuing,  highly  capable,  multidisciplinary 
team  with  access  to  all  of  the  relevant 
information.  The  C4ISR  Decision  Support 
Center  is  one  model  of  a  government-run 
organization.  Another  potential  organizational 
model  is  the  Phase  One  Engineering  Team 
(POET^,  which  draws  on  personnel  from 


FFRDCs  and  not-for-profits  to  support  analyses 
of  ballistic  missile  defense  issues. 

5.4  Some  Encouraging  Signs.  In  spite  of  the 
many  deficiencies  in  the  C4ISR  anaJysis  area, 
there  are  several  encouraging  signs.  First,  the 
analysis  community  realizes  that  there  are  many 
deficiencies  and  that  steps  need  to  be  taken  to 
remedy  the  deficiencies.  High-level  decision 
makers  are  aware  of  the  deficiencies,  and  they 
seem  willing  to  invest  the  resources  to  address 
the  problems. 

Secondly,  efforts  are  already  underway  to 
remedy  some  of  the  deficiencies.  For  example, 
JWARS,  NSS,  NETWARS  and  the  C4ISR 
Federation  are  all  major  programs  to  develop 
improved  decision  support  tools.  The  M&S 
community  is  beginning  to  understand  that  a 
new  paradigm  in  modeling  is  necessary  with 
explicit  consideration  given  to  modeling  the 
information  and  decision  processes,  joint 
operations,  communications,  fusion  and 
perception  management.  However,  this 
workshop  has  pointed  out  that  no  single 
framework,  no  single  tool,  no  single  measure 
was  sufficient  to  deal  with  the  full  set  of  issues 
of  interest.  The  challenge  is  to  select  and 
orchestrate  the  needed  tools  and  measures  to 
reflect  the  substantive  issues  in  question  and  to 
be  responsive  to  the  needs  of  the  decision 
makers. 
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Major  Theater  War: 

Working  Group  Agenda 

•  Tuesday  Afternoon: 

—  1300-1430  Introductions  and  Opening  Remarks  (3-5  Min. 
per  person) 

—  1500-1530  Chair  stakes  out  mission  and  ROE 

—  1530-1630  Problem  Articulation  For  Next  Day 

•  Wednesday  Morning 

—  0800-0900  Analysis  of  the  Current  State  of  the  Art 

—  0900-1000  New  Methodologies  and  Tools 

—  1000-1100  Requirements  for  C4ISR  Analysis 

•  Wednesday  Afternoon 

—  1230-1330  Synthesis  of  Morning 

—  1330-1600  Recommendations 

—  1600-1630  Wrap-up 

•  Thursday  Morning 

—  0800-1000  Recap/  New  Topics 

—  1000-1200  Write  Up  Findings  As  Group 


Major  Theater  War: 

Approach 

•  What  is  the  Scope  of  this  Working  Group? 

—  Definitions 

•  What  is  the  current  State  of  the  Art? 

—  Models 
—  Analysis 

•  What  are  the  appropriate  tools  to  “give  DoD  the  capability  to  do  C4ISR 
Analysis?” 


Approach: 

The  approach  detailed  on  this  and  the  next  slide  were  developed  beforehand  by  the 
working  group  co-chairs.  The  group  agreed  to  follow  this  general  approach. 
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Major  Theater  War 

Approach  (confd) 


What  are  the  requirements  for  C4ISR  Analysis? 

—  Models 
—  Data 

—  Analysis  architectures 
—  Supporting  infrastructure 

What  are  the  recommendations  for  the  Workshop  Sponsors? 

The  Bottom  Line 
—  Outbriefing 
—  Final  report 


Major  Theater  War: 

Scope  of  the  Working  Group 

•  Consider  issues  related  to  producing  analysis  in  support  of  major  DoD 
programmatic  decisions  (e.g.,  QDR) 

-  Analysis  of  C4ISR  programs  and  issues  (e.g.,  CONORS, 
interoperability) 

-  Analysis  of  other  programs  and  force  structure  in  JV  2010,  to  include 
shortcomings 

•  Examine  how  we  can  do  the  analysis,  which  includes,  but  is  much  more  than, 
developing  models  of  C4ISR 


Scope  of  the  Working  Group: 

The  first  substantive  task  of  the  group  was  to  reach  a  consensus  on  what  matters  were 
properly  within  the  purview  of  the  group,  cind  what  matters  were  not.  The  group  agreed 
that  the  focus  would  be  on  the  use  of  C4ISR  analysis  and  tools  to  support  major 
programmatic  decisions  such  as  the  QDR.  This  would  include  the  analysis  of  C4ISR 
systems  in  the  context  of  total  force  structure,  but  would  also  include  the  representation 
of  C4ISR  in  the  analysis  of  force  structure  and  military  sufficiency,  again  in  the  context 
of  QDR-like  analysis.  The  proper  use  of  C4ISR  analysis  and  models  in  support  of  CINC 
COA  determination  (in  particular  CENTCOM  and  USFK  MTW  modeling)  was 
considered  to  be  outside  of  the  scope  of  the  group. 

Some  particular  comments  and  issues  raised  by  the  group  in  this  discussion  were: 

•  Is  an  MTW  scenario  enough  to  make  statements?  The  DPG  IPS  are  useful  to  size  the 
force  structure  but  not  necessarily  the  ISR  component;  ISR  issues  usually  require 

fleshing  out”  the  scenarios.  The  DPG  IPS  are  useful  to  size  the  force  structure  but 
does  not  necessarily  cover  totality  of  capabilities  required 

•  There  is  a  need  to  model  Red  behavior  in  as  much  detail  and  fidelity  as  Blue 
behavior. 

•  Who  is  the  Decision  Maker?  In  general,  the  SECDEF/Congress  (e.g.,  QDR),  but 
CINC  requirements  were  included  in  subsequent  discussions.  Specific  Decision 
maker  is  usually  the  DepSECDEFA^CJCS. 
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Major  Theater  War: 

Current  State  of  the  Art  (Summary) 

•  Legacy  Combat  Models 

—  Scripted  C2,  some  rule  set  behavior 
—  Some  communications  but  simplistic 
—  Little  effect  shown  from  attacking  C2 
-  Deterministic  intelligence  with  little  quality  reference 
—  Limited  explicit  representation  of  C4ISR  systems  and  mapping  into 
combat  outcome 

—  Sensor  collection  models  may  be  adequate 
—  Intel  and  Combat  models  stovepiped 


Current  State  of  the  Art  (Summary);  (Slides  7  and  8): 

These  two  slides  were  prepared  beforehand  by  the  co-chairs.  There  was  some  discussion 
and  disagreement  over  specific  items  on  the  slide,  but  the  group  agreed  to  the  words  that 
appear  on  this  final  version.  Some  specific  comments: 

•  Current  (legacy)  models  aren’t  all  bad;  modeling  and  analysis  work  to  date  have  some 
useful  attributes.  However,  in  general,  the  state-of-the-art  is  woefully  lacking  in 
representation  of  C3  issues.  ISR  collection  attributes  are  fairly  well  understood.  Most 
importantly,  the  cause  and  effect  of  information  on  battle  are  not  well  understood  or 
treated  with  existing  models. 

•  It  is  necessary  to  differentiate  between  a  single  model  and  a  federation  of  models.  The 
advantages  and  disadvantages  of  various  approaches  are  discussed  in  later  slides. 

•  Battle  outcome  as  function  of  maneuver  (e.g.,  flank  attacks)  are  not  well  treated  or 
understood. 

—  The  impact  of  information  on  C2  function  is  currently  based  on  timeliness;  the 
quality  of  the  information  available  for  changing  decisions  is  not  considered. 

—  In  legacy  models,  doctrine  is  hardwired. 
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Red  C2  is  needed  to  be  reactive  to  Blue  decisions,  to  show  impact  of  Blue  C4ISR. 

The  impact  may  be  dampened  by  red  “inertia”  in  present  analysis  (i.e..  Red 
follows  same  plans  regardless  of  Blue  capabilities). 

Conununications  physics  is  understood  and  can  be  modeled  if  desire  is  there. 
Communications  are  usually  ignored  because  of  the  laborious  and  time 
consuming  issues  associated  with  the  details  of  such  efforts.  The  community  has 
ignored  the  impact  of  these  assumptions. 
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Major  Theater  War: 

Current  State  of  the  Art  (Summary)-conCd 

*  Campaign  models  under  development  -  still  no  theory 
—  Still  much  scripting 

—  Using  present  models  as  basis,  we  stUl  don’t  have  good  C4ISR  process 
representations  in  many  areas 
—  Rule  set  behavior 

—  Stochastic  intelligence  but  simplistic  fusion 
—  Mixed  effect  shown  from  attacking  C2 
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Major  Theater  War: 

State  of  the  Art 

•  Collection  Modeling 

-  IMINT  — Good 

—  ELINT  —  Poor  due  to  emissions  not  well  modeled  (scenarios) 

—  COMINT - Scripted  to  Zero  (emissions) 

—  Cross  Cueing  —  Poor 

-  MTI  — Marginal  (Red  Behavior) 

—  MASINT  —  Non-imaging  IR  OK,  the  remainder  non-existent 

-  HUMINT  —  Humans???? 

•  PED  Models 

—  Good  representation  of  time  and  volume  flows 
—  Poor  representation  of  content  (information  vice  data) 

—  Dissemination  and  representation  of  organizational  (hierarchy) 
knowledge  poorly  represented 

•  Fusion  Models  -  non  existent 


The  State  of  the  Art: 

The  group  decided  that  it  would  be  useful  to  break  up  the  ISR  and  C4  process  in  order  to 
assess  specific  capabilities  and  deficiencies  in  the  state-of-the-art  representations  of  this 
process  in  current  models.  The  following  diagram  was  used: 


Collection 

PED 

Strike 


Maneuver 


The  assessment  was  then  made  on  various  aspects  of  the  C4ISR  process  within  each  box. 
In  general,  collection  modeling  was  regarded  as  good  overall  (with  specific  deficiencies 
in  some  areas  such  as  HUMINT,  for  example).  PED  models  of  total  flow  are  mature  but 
lack  consideration  of  the  quality  of  information,  which  is  essential  to  answer  the  “so 
what?”  response  to  volume  measures.  Fusion  models  are  essentially  non-existent,  in  real 
world  as  well  as  in  analytic  modeling.  Obviously,  some  correlation  and  filtering  is  done 
in  the  real  world  but  this  automated  process  is  regarded  as  immature. 
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Major  Theater  War: 

State  of  the  Art  (confd) 

*  Decision  Modeling 

—  Rule  set  construction  well  understood 

—  What  rules  to  use,  especially  in  the  future  (different  doctrine, 

CONOPS)  not  available 

—  Human  factor  issues  not  understood 

—  What  decisions  to  model?  Doctrine,  “real  world”,  normative,  optimal? 

—  Strike  decisions  /  resource  allocation  relatively  well  understood, 
maneuver  issues  and  effects  not  well  understood 

•  Perception 

—  COP  representation  does  not  reflect  quality  of  information,  the  degree 
to  which  different  sources  agree/disagree,  or  alternative  hypotheses 

—  If  the  COP  has  an  area  with  no  Red,  is  that  because  we  have  extensively 
observed  the  area  and  seen  no  Red,  or  is  it  because  the  area  is  obscured 
from  view,  or  is  it  because  we  haven’t  looked  there? 


The  State  of  the  Art  (Continued): 

Decision  and  perception  modeling  are  also  immature.  Even  if  the  process  of  representing 
a  decision  is  known  (e.g.,  rule-based  inference),  what  decisions  to  use  is  up  in  the  air. 
Unfortunately,  the  choice  of  decisions(s)  to  use  in  models  often  dominates  the  model 
outcomes,  having  an  effect  much  greater  than  that  realized  by  changes  in  systems  or 
forces.  Do  we  have  the  data  to  determine  when  decisions  are  made  and  how  good  they 
would  be?  Decisions  should  be  evaluated  as  a  function  of  risk  based  on  the  amount  of 
information  available,  not  necessarily  the  outcomes  (hindsight).  It  is  possible  to  make 
decisions  that  are  “bad”  from  the  view  of  the  outcome  but  “good”  with  respect  to  the 
information  available  at  the  time  that  the  decision  was  made. 
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Major  Theater  War: 

State  of  the  Art  ( cont  ) 

*  Communications 

—  Communications  network  modeling  relatively  well  understood 
—  Physics  well  understood  but  not  necessarily  included 
—  Bandwidth  and  legal  issues  not  represented 
—  Time  and  volume  analysis  moderately  well  understood 

—  Often  don’t  do  the  communication  modeling  we  understand  because  of 
time  and  data  constraints 

—  Often  done  off-line  but  this  inhibits  dynamics 

—  Difference  between  passing  data  (connectivity,  delay)  and  passing 
information  not  understood 

-  Impact  of  loss  of  communications  on  C4ISR  (which  in  turn  effects 
battle  outcomes)  not  well  understood 

•  Planning  /  Re-planning 

—  Plan  execution  and  development  relatively  well  represented 
—  Re-planning  poorly  represented  if  at  all 

The  State  of  the  Art  (Continued): 

Communications  modeling  (physics  based)  is  well  understood;  the  modeling  of 
information  flows  (vice  bits  per  second)  is  not.  Automated  planning  and  re-planning 
capabilities  are  also  lacking.  ISR  (collection)  planning  is  easier  because  much  is 
preplanned,  but  on  the  fly  NAI’s  are  not  done. 
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Major  Theater  War; 

State  of  the  Art  (confd) 

•  Combat  Outcomes 

—  Strike  relatively  well  understood 

•  BDA  and  re-planning  not  as  well  understood 
—  Maneuver  not  well  understood 

•  Even  if  we  do  C4ISR  well,  impact  may  not  be  reflected.  E.g.,  what 
if  we  attack  Red  C4ISR  and  surprise  him?  What  is  the  effect  of 
surprise? 

•  Representing  the  effect  of  maneuver  on  Red  behavior  not 
understood  (e.g.,  causing  Red  to  withdraw  without  actually 
attacking  him). 

—  Total  Force  Protection  and  Focused  Logistics  are  buzzwords  with 
respect  to  C4ISR  impacts 

•  Limited  functions  (e.g.,  TMD)  are  reasonably  well  represented 


The  State  of  the  Art  (Continued): 

Basic  research  is  still  needed  in  basic  combat  effects  modeling.  C4ISR  capabilities  should 
enhance  maneuver,  for  example,  but  the  effect  of  that  maneuver  is  not  well  represented  in 
current  attrition-based  models.  There  are  two  issues  here.  Our  basic  inability  to  tie  Situational 
Awareness  to  a  decision  in  a  predictive  manner  means  that  we  can  not  then  link  the  decision 
to  a  battle  outcome.  The  second  issue  is  that  most,  if  not  all,  ground  combat  models  are 
insensitive  to  time  and  space  of  attack  and  what  they  mean  to  success.  Therefore  we  can 
neither  predict  accurately  what  a  commander  would  do  differently  as  a  result  of  information 
or  show  how  that  difference  in  movement  or  timing  resulted  in  a  change  to  his  probability 
of  success.  As  long  as  battle  outcome  metrics  (casualties,  force  exchange  ratios,  etc.)  are 
used  to  measure  the  effectiveness  of  new  systems  (to  include  C4ISR  systems)  and  forces, 
better  representations  of  basic  combat  effects  will  be  required.  Some  specific  observations: 

•  Models  are  insensitive  to  outcome  based  on  first  order  effects;  this  is  much  more 
acute  in  maneuver 

•  There  is  little  cross  representation;  e.g.  Strike  and  lO  impact  on  maneuver. 

•  Legacy  Models  have  good  physics  for  movement  and  for  some  outcomes. 

The  bottom  line: 

•  Campaign  models  are  desired  by  decision  makers  —  MOFEs  are  preferred. 

•  It  is  necessary  to  move  away  from  FER  to  representing  maneuvering  effects  logistics, 
strategic  effects,  time  to  achieve  objective,  fratricide,  10,  etc. 

•  Current  models  all  have  significant/critical  deficiencies  for  C4ISR  analysis;  some  have 
partial  capabilities 

•  Major  problems  lie  in  combat  modeling  as  well  as  in  C4ISR  models  and  integration 
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Other  areas  of  deficiencies: 

•  Non-Combat 

•  Red  C4ISR 

•  Red  ROE  decision  heuristics 

•  10 

•  Other  Coalitions 


Major  Theater  War: 

Requirements  for  C4ISR  Analysis 

•  Analysis  Set  Up 

—  Problem  Deflnition 
—  Experimental  Design 
—  Risk  and  Uncertainty 
—  Tools 
—  Scenarios 
—  Assumptions 

•  Data 

—  Sources 

—  Assumptions  on  data 
—  Aggregation 
—  Missing  Data 
—  Incorporating  Judgement 
-  W&C 


Requirements  for  C4ISR  Analysis:  (slides  13  and  14) 

These  slides  are  just  meant  to  set  the  stage  for  those  topics  that  we  discuss  in  detail  over 
the  next  10  slides. 
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Major  Theater  War: 

Requirements  for  C4ISR  Analysis  (cant 


Tools  and  Methodologies 

-  Single  campaign  model  (e.g.,  TACWAR,  JWARS) 
—  Hierarchy  of  Models 

-  Federation  of  Models 
—  Non>Simulation  Tools 


Major  Theater  War: 

Analysis  Setup 

•  Problem  Dennition 

—  More  is  Better? 

-  DeHned  Objectives 

-  Constraints?  Casualties 
—  Time  and  Resources 

•  Types  of  questions 

—  What  is  best  among  alternatives? 

—  How  much  better  is  A  than  B? 

—  How  much  A  do  I  need? 

—  What  is  the  tradeoff  between  A&B? 

—  How  does  A  effect ....? 

—  How  much  does  A  cost? 

-  How  well  does  A  perform? 

—  Identify  Offsets 

—  What  is  the  marginal  return  of  A  in  combat  metrics? 
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Major  Theater  War: 

Problem  Definition  -  Drivers 


•  Time 

•  Resources 

—  People 
—  Money 
—  Available 
Model 

•  Question 

•  Politics 

•  DM  Expectations 

•  DM  Focus 


•  Security 
Considerations 

•  Coalition 
Considerations 

•  Data  Availability 

•  Scenario  Availability 

•  Culture 

•  Previous  Studies 


Problem  Definition: 


Some  of  the  most  critical  issues  surrounding  studies  of  this  magnitude  have  little  to  do 
with  what  may  be  called  the  classical  design  features  of  the  scientific  method.  Instead, 
they  revolve  around  completely  exogenous  factors  that  frequently  end  up  determining  the 
results  out  of  hand.  These  variables  include  the  time  allowed  to  complete  the  effort,  the 
resources  provided,  the  politics  of  both  the  question  to  be  answered  and  the  organizations 
chosen  to  run  it  (as  well  as  those  left  out),  and  the  decision  maker’s  expectations 
surrounding  results.  These  issues  in  a  strict  scientific  sense  should  have  no  bearing  on 
results  and  yet  they  frequently  are  the  first  order  effects  that  determine  the  answer. 

Of  the  remaining  issues  the  concepts  of  clearly  articulating  the  question  and  working  with 
the  decision  maker  to  insure  that  the  question  is  properly  understood  by  all  and  framed 
such  that  an  answer  can  be  derived  is  the  most  meaningful  part  of  the  analytical  process. 
Scenarios,  data  and  past  work  are  all  acknowledged  as  both  required  and  unevenly 
provided  and  exercised  in  most  study  efforts. 
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Major  Theater  War: 

C4ISR  MOEs 


•  Definition  of  C4ISR  MOPs  and  Battle  MOEs  not  the  problem. ...  It  is  the 
mapping  of  C4ISR  MOPs  to  Battle  MOEs 

•  Very  Dependent  on  Context 

•  Analysis  requires  multiple  MOEs 


MOEs: 


The  definition  of  C4ISR  MOPs  is  not  a  problem  within  the  community  from  the 
perspective  of  the  group.  Nor  is  there  difficulty  in  establishing  criteria  for  mission 
success,  e.g.  MOEs.  The  real  issues  continue  to  be  the  ability  of  the  analytical 
community  to  confidently  establish  direct  linkages  between  the  C4ISR  system 
performance  measures  and  the  criteria  for  mission  effectiveness. 

It  is  noted  that  no  single  MOP  or  MOE  satisfies  as  a  global  metric  for  performance  or 
effectiveness  across  all  mission  areas,  nor  is  it  sound  analytical  practice  to  only  measure  a 
single  attribute  as  the  criterion  in  a  study  effort.  In  studies  as  large  and  complex  as  those 
involved  with  MTW  multiple  criteria  are  required. 
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Major  Theater  War: 

Scenario  and  Excursions 

Deployment  Timeline  /Force  Structure  for  C4ISR  assets  not  represented 
Baseline  Typically  Directed 
—  Problems: 

•  Multiple  Detailed  Versions  of  DPG  Scenarios  Exist 

•  Multiple  Predictions  of  Future,  Conflict  Areas 

•  Little  DeHnition  of  Red/Allied  beyond  initial  lay  down 
Should  we  vary  Scenarios?  How  much? 

Red  Scenario  should  be  dynamic  not  fixed 


Scenarios  and  Excursions: 


There  are  two  issues  embodied  in  this  slide.  The  first  issue  is  the  fact  that  rarely,  if  ever, 
are  C4ISR  asset  movements  into  theater  accounted  for.  The  second  issue  is  that  high  level 
scenarios  such  as  the  DPG  do  not  specifically  address  the  totality  of  the  variables 
associated  with  setting  up  an  analysis. 

The  first  issue  results  in  unrealistic  assumptions  about  C4ISR  assets  in  theater  early  in  a 
campaign  and  because  of  this  likely  overstate  performance.  The  second  issue  requires  the 
analyst  to  create  portions  of  the  scenario  for  a  particular  effort  (e.g.  specific  force 
laydown,  enemy  behavior,  own  force  ROE,  etc.).  All  of  these  assumptions  result  in  a 
multiplicity  of  scenarios  eventually  being  created  from  what  is  believed  by  senior  levels 
to  be  a  homogenous  baseline  from  which  all  efforts  departed. 

Other  concerns  revolve  around  the  issues  of  whether,  in  today’s  environment,  it  is  more 
applicable  to  analyze  full  blown  MTW  events  rather  than  multiple  Smaller  Scale 
Regional  Contingencies. 
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Major  Theater  War: 

Experimental  Design 

•  Classical  designs  often  not  useful  because  statistical  model  not  appropriate 

•  Too  many  things  that  should  be  varied  -  judgement  on  what  needs  to  be 
varied 

•  Answers  tend  to  be  multidimensional;  combination  (via  MAU?)  often  not  well 
defined  or  explained  to  DM 

•  Exploratory  analysis  and  prescreening  may  be  useful 

•  Context  dependent 


Experimental  Design 

A  review  of  recently  completed  analyses  made  in  the  context  of  an  MTW  shows  that 
basic  problems  exist  in  terms  of  the  sheer  number  of  independent  variables  that  can 
impact  results.  This  “overflow  of  dimensionality”  results  in  an  experimental  design  that 
is  unmanageable.  The  way  recent  efforts  have  dealt  with  the  issue  is  to  somewhat 
arbitrarily  reduce  the  number  of  variables  for  which  the  analysis  will  study  the  impact.  In 
so  doing,  the  analytical  team  runs  the  risk  of  not  completely  exploring  the  cause  and 
effect  relationships  and  dependencies  between  multiple  variables.  The  trade-off  for  this 
is  obviously  time  and  money.  In  general  the  group  felt  that  more  research  needed  to  go 
into  how  to  effectively  manage  this  effort  and  maybe  research  that  would  provide  insight 
into  the  variable  dependencies  that  would  allow  constrained-run  matrices  in  the  future. 

Unfortunately,  many  analysts  lack  an  appreciation  for  the  statistical  issues  around  the 
experimental  design  and  interpretation  of  analysis  result.  For  example,  many  of  the 
packages  and  statisticzd  techniques  rely  on  assumptions  such  as  normality  and 
homoscedasticity,  which  are  usually  not  tested  for.  Another  example  is  the  dearth  of 
statistical  tests  for  ordering  means  (vice  determining  if  they  are  different). 
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Major  Theater  War: 

Data 

•  Few  Accepted  Sources  for  C4ISR  Data 

•  Red  C4ISR/Behavior  data  lacking 

•  Human  Performance  data  lacking 

•  Hard  to  generate  data  for  poorly  understood  models 

•  Future  systems  data  lacking 

•  Existing  models  haven’t  generated  requirements  for  this  type  of  data, 
therefore  no  one  is  tasked/funded  to  gather  it 

•  Recording/cataloguing  data  from  other  analyses  is  necessary 


Data: 

The  discussion  here  centered  on  the  distinctions  between  system  performance  data^ 
scenario  data,  and  “behavioral”  data  (e.g.  human  decision  making).  Although  there  are 
always  issues  of  how  to  characterize  future  systems  and  difficulties  in  getting  some 
organizations  (e.g.,  NRO)  to  share  data,  the  group  felt  that  MTW  was  the  most  mature 
area  of  data  collection  and  storing  in  the  community.  There  are  several  organizations  that 
are  chartered  to  collect  and  hold  such  information  for  general  use  and  it  is  also  the  most 
easily  understood  data  within  the  non-modeling  community.  The  next  layer  of  data, 
scenario  data,  has  many  problems  associated  with  it.  As  alluded  to  above,  multiple 
analyses  seem  to  beget  multiple  interpretations  of  proscribed  “baselines”.  In  addition  this 
type  of  data  is  incomplete  in  its  representation  of  red  forces  especially  in  terms  of  Red 
C4ISR  systems  and  Red  Behavioral  data.  The  final  and  most  vexing  area  of  data  required 
is  that  on  behavioral  entities.  There  are  virtually  no  reliable  data  sources  for  C2  events 
that  require  decision  making  to  occur  or  data  that  suggests  what  courses  of  action  may 
transpire  from  equivalent  battlefield  events  (Red  or  Blue).  The  group  felt  that  in  order  to 
advance  the  science  of  C4ISR  modeling  much  research  was  required  here,  followed  by 
the  nomination  of  a  group  or  organization  to  manage  and  store  the  results. 
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Major  Theater  War: 

Data  (Continued) 


•  Existing  data  doesn’t  necessarily  map  into  DPG  scenarios 

•  Big  difference  between  physical  data  and  behavioral  data 

•  Background  Data  (civilian  traffic,  etc.)  not  available  “Baseline”  Data  hard  to 
deffne  when  C4ISR  changes  can  change  scenario,  decisions,  behaviors 

•  Scenario  Data  developed  for  unique  modeling  constructs 

•  Security  Issues 

•  Politics/Data  Sharing 
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Major  Theater  War: 

Methodology /T ooh 

Monolithic  Single  Camnaign  Model 

Effective  Use: 

Problems: 

•  Calculates  MOFE 

•  Theoretically  Flawed 

•  Full  context  of  conflict 

•  Tend  to  focus  on  attrition 

•  Demonstrate  what  you 

•  Very  complex  and  data  intensive 

already  decided 

•  We  are  building  a  tool  based  on 

•  OP  Plan  Analysis 

today’s  understanding  of  C4ISR 

•  Combine  with  higher-level 

impacts,  while  no  one  really 

models  to  explore  particular 

knows  the  true  mechanisms  by 

issues 

which  C4ISR  impacts  battle 

•  V&V  with  evolving  models 

Models  and  Tools: 


The  group  examined  four  major  approaches  to  representing  C4ISR  systems  in  the  context 
of  an  MTW  campaign  analysis.  These  approaches  were: 

•  Developing  a  single,  monolithic  campaign  model  that  represents  “everything  ” 

•  Developing  a  hierarchy  of  models, 

•  Developing  a  federation  of  models,  and 

•  Using  nontraditional  techniques  outside  of  simulation. 

Pros  and  cons  of  each  approach  were  discussed. 

Monolithic:  There  are  two  major  points  that  need  to  be  brought  out  when  looking  at  large 
campaign  level  monolithic  models.  The  first  is  that  they  are  by  their  nature  low  fidelity 
representations  of  the  combat  and  combat  interactions,  as  such  it  is  extremely  hard  to 
identify  cause  and  effect  relationships  for  those  features  that  are  not  explicitly  designed 
into  the  model  as  a  major  driver  of  combat  results.  Given  that  this  has  never  been  done 
for  C4ISR  it  is  next  to  impossible  to  use  such  models  today  to  adequately  address  the 
impact  of  information.  The  second  issue  is  that  it  will  only  be  possible  in  the  future  if  the 
theory  of  C4ISR  impact  on  battle  operations  is  canonized  and  put  into  these  models  as  a 
first  order  driving  effect. 
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Another  observation  not  well  appreciated  by  senior  decision  makers  is  the  fact 
that  current  C4ISR-based  advanced  modeling  efforts  (e.g.,  JWARS  and  JSIMS)  will  not 
provide  any  better  representation  of  C4ISR  systems  and  processes  that  the  current  (poor) 
state-of-the-art  allows  unless  research  (vice  coding)  is  invested  in. 
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Major  Theater  War: 

Methodology /Tools  { Continued) 

Hierarchy  of  Models 

Effective  Use; 

Problems; 

•  Data  generation  for  low-res 

•  Aggregation/Dissaggregation 

models 

•  Model  treatments  often 

•  Determine  what  issues  to 

inconsistent 

look  at 

•  Confidence  Intervals 

•  “Flesh  Out”  detailed  issues 

...Accuracy  measures  across 

•  Cross-check  ‘S^alidity” 

multiple  models 

•  Depth  and  Breadth 

•  Choosing  what  to  represent  at 

higher  resolution 

•  Multiple  answers  to  DM 

Hierarchy: 


This  approach  has  one  main  draw  back.  Data  is  usually  modeled  at  different  levels  of 
fidelity  across  the  continuum  of  the  modeling  structure.  Whether  it  is  the  data  used  to 
created  the  scenario  or  the  data  used  in  treating  impacts  of  various  attributes  of  the  war 
they  are  probably  considered  in  differing  degrees  of  fidelity  and  therefore  the  linkages  of 
models  becomes  suspect. 
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Major  Theater  War: 

Methodology /Tools  (  Continued ) 

Federation  of  Models 


Effective  Use: 

*  Accepted  Models  do  Exist 

*  Treats  Individual  Functions  in 
Higher  Fidelity  than  Campaign 
model  allows 

*  Insures  Cause  and  Effect 
Explicitly  Treated  and 
Understood 

*  Data  generation  for  Combat 
Models 

*  “Flesh  Out”  detailed  issues 

*  Cross-check  “validity” 

*  Depth  and  Breadth 


Problems: 

•  Target  Domain  mapping 

•  Aggregation/Dissaggregation 

•  Time  and  Space  Discontinuities 

•  Model  treatments  inconsistent 

•  Feedback  loops  lacking 

•  Hard  to  ID  Synergy 

•  Missing  Pieces 

•  Multiple  answers  to  DM 

•  VV&C 

•  Abuser  Friendly 


Federation: 


To  date  this  has  been  the  most  successful  method  for  capturing  the  cause  and  effect 
relationships  implicit  in  C4ISR  on  the  battlefield.  It  allows  fidelity  where  fidelity  is 
needed  and  allows  the  analytical  team  to  review  the  cause  and  effect  relationships  as  they 
unfold  at  the  seams  of  the  individual  models.  These  types  of  modeling  efforts  still  fall 
prey  to  the  same  issues  discussed  above  for  Hierarchy  of  models. 
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Major  Theater  War: 


Non-Simulation  Approaches: 


•  Spread  Sheets 

•  Analytic  Model 

•  Man-in-The-Loop 

•  War  games 

•  Exercises 

•  Historical  Analysis 

•  Surveys/Operator  Observations 


Non-Traditional  Techniques: 


A  variety  of  non-simulation  approaches  may  be  used  to  answer  parts  of  the  analysis 
question  (although  it  did  not  appear  that  any  single  technique  was  robust  enough  to 
approach  answering  all  C4ISR-related  questions  in  an  MTW  context).  These  techniques 
are  listed  on  the  slide,  but  should  be  considered  to  be  a  representative  sample  of  the 
techniques  available  rather  than  a  comprehensive  list. 


Major  Theater  War: 

Assessm&ftt  of  Currenf  Stote-of-the-Pmctice 


Assessment  of  the  State-of-the-Practice:  Using  the  same  chart  as  used  by  other  working 
groups: 


Problem  Structuring  (Yellow): 

•  Strike  issues  relatively  well  understood,  maneuver  issues  poorly  understood 

•  Link  between  C4ISR  MOEs  and  MOFEs  not  well  understood 

•  System  level  performance  issues  &  MOEs  relatively  well  understood 


Human  Factors  and  Organization  (Red) 

•  Behavior  models  not  developed 

•  No  Red  behavior  models,  data,  or  dynamic  scenarios  available 

•  Data  for  behavior  and  decision  making  not  available 

•  Human-in-the-Loop  approaches  are  not  repeatable  and  may  not  be  verifiable 

•  Do  we  model  doctrine,  “real-world,”  normative  or  optimal  decisions? 

•  Decisions  tend  to  cause  significantly  nonmonotonic  outcomes.  They  also  tend  to  be 
nonlinear  with  respect  to  the  stimulus  (e.g.,  thresholds) 


Assessment  of  the  State-of>the-Practice  (contM): 


Scenarios  (Red) 

•  DPG  scenarios  really  don’t  get  at  C4ISR  issues 

•  Conflict  between  a  defined  (scripted)  scenario  and  showing  C4ISR  effects  (which  cause  dynamic 
effects) 
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•  Red  behaviors  (especially  dynamics)  not  defined  in  scenarios  or  supporting  data 
For  ISR,  DPG  /  MTW  scenarios  may  not  be  the  most  stressful  condition  (DCI/peacetime  may  be 
largest/most  stressing  requirements) 

•  Issues  that  have  great  impact  on  C4ISR  may  not  be  significant  in  the  scenario  (e.g.,  SCUDs  that 
launch  against  political  vice  military  targets) 

•  Decisions  &  doctrine  not  defined  in  future  scenarios 

Measures  of  Merit  (Yellow) 

•  C4ISR  MOEs  and  MOFEs  well  understood  but  linkage  between  them  is  not 

•  MOMs  very  dependent  on  context  —  hard  to  make  any  general  statements 

•  A  good  decision  is  not  synonymous  with  a  good  outcome.  This  creates  a  problem  when  you  use 
MOFEs  as  criteria  for  ev^uation 

Tools  and  their  Applications  (Red) 

•  No  good  models  or  tools  exist  for  many  C4ISR  processes 

•  Red  C4ISR  not  well  understood  or  modeled 

•  Models  under  development  (e.g.,  JW ARS)  are  taking  the  “best  current  modeling  approach”  as  their 
basis  —  since  no  good  models  exist  currently,  JWARS  will  not  have  any  models  to  incorporate. 

•  Hierarchical  and  federated  modeling  approaches  have  significant  problems  with  model  consistency 

•  You  can’t  model  well  what  you  don’t  understand 

Data  (Red) 

•  Data  generally  does  not  exist  on  behavior  and  human  issues 

•  Red  C4ISR  data  particularly  lacking  (especially  behavior) 

•  Doctrine  is  a  form  of  data  for  C4ISR  models  —  yet  we  don’t  have  that  well  quantified,  especially 
when  future  systems  are  included 

•  There  is  a  big  difference  between  physical  attribute  data  and  behavioral  /  decision  data 

•  Funding  is  not  there 

Risk  and  Uncertainty 

•  Will  the  systems  perform  as  expected?  Relatively  well  understood 

•  Will  the  systems  effect  combat  as  modeled  /  hypothesized?  Not  understood 

•  Is  the  context  correct?  Not  well  defined  for  C4ISR  —  e.g.,  doctrine,  future  baseline,  data,  etc. 

•  Uncertainty  is  often  modeled  with  probability —  but  we  don’t  understand  the  models,  much  less  the 
distributions,  much  less  the  distributional  parameters 

Report  (Red) 

•  Because  we  lack  C4ISR  models  and  data  are  lacking,  many  assumptions  have  to  be  made.  These 
assumptions  (and  their  impact)  may  not  be  well  reported  to  and  understood  by  the  DM 

•  Linkage  assumptions  between  C4ISR  and  combat  outcomes  often  not  reported  (may  not  even  be 
understood  by  analyst  if  implicit  in  models) 
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Breakout  of  the  state-of-the-art  within  the  intelligence  cycle: 


These  ratings  were  discussed  previously;  this  slide  simply  summarizes  the  ratings. 
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Major  Theater  War: 

Recommendations 

•  Research  needed  to  how  we  reflect  C4ISR  outcomes  into  military  effects  (e.g., 
maneuver) 

•  Research  needed  on  perception  development  and  fusion 

•  Research  and  data  needed  on  decision  making  (C2)  representation 

•  Funding  needed  for  data  collection  and  management.  Cannot  just  be  tied  to 
existing  models.  (Will  save  analysis  money  in  long  run) 

•  We  need  to  explain  better  to  DM  where  model  and  analysis  limitations  are. 
Analysis  demands  seem  to  have  shorter  timelines  and  less  money.  “Quick 
turnaround”  analysis  may  require  extensive  preparation  (scenarios,  data, 
etc.)  be  accomplished  beforehand. 

•  New  model  development  (e.g.,  JWARS)  is  based  on  “using  the  best  of  the 
accepted  algorithms.”  Since  C4ISR  algorithms  are  missing  now,  the  next 
generation  of  models  will  not  have  proper  C4ISR  representations. 
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Major  Theater  War: 

Recommendations  (Cont  ^d) 

•  Need  to  develop  knowledge  base  for  Red  behavior  (not  just  systems  and 
general  doctrine).  Coalition  behavior  information  also  needed. 

•  JDS  needs  specific  tasking  to  support  C4ISR  analysis  beyond  JWARS.  This 
will  require  additional  funding. 

•  Intel  scenarios  and  combat  scenarios  need  to  be  matched.  Non-tactical  intel 
requirements  need  to  be  defined  in  context  of  campaign. 

•  To  ensure  consistency  across  services,  the  QDR  scenarios  need  to  be 
fleshed  out  now  by  (or  managed  by)  a  single  joint  agency.  This  includes 
data  and  Red  representation. 

•  More  data  sharing  and  sanitization  of  data  (e.g.,  national  systems)  needed. 
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Major  Theater  War: 

The  Bottom  Line 

•  Insight:  Basic  research  is  needed.  Some  major  areas  (not  all  inclusive): 

—  Linkage  between  C4ISR  effects  and  campaign  outcomes, 

-  Fusion,  perception  development  and  decision  making, 

—  Basic  combat  itself  (e.g.,  maneuver  effects). 

•  Insight:  JWARS  and  other  new  model  efforts  are  incorporating  the  “best” 
current  models  -  this  approach  will  not  work  with  respect  to  C4ISR. 

•  Recommendation:  Develop  prioritized  list  in  the  areas  of  C4ISR  effects  on 
combat,  C2,  ISR,  communications  and  basic  combat  to  direct  the  research 
plan.  Action:  J-2,  J-6,  J-8,  DSC  (J-6  lead).  Fund  research.  Action: 
DD(R&E)  /  ASD(C3I).  This  research  needs  to  be  implemented  into  new 
model  developments,  but  we  do  not  need  the  effort  focused  on  a  particular 
tool. 


Bottom  Line”  Recommendations: 

The  Bottom  Line”  slides  include  the  Working  Group’s  recommendations  regarding  the 
appropriate  implementing  organizations.  Those  recommendations  have  not  been  formally 
coordinated  with  the  organizations  listed. 
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Major  Theater  War: 

The  Bottom  Line  (Cant'd) 


•  Insight:  There  are  fundamental  data  gaps  to  support  C4ISR. 

•  Recommendation: 

-  Enumerate  the  data  requirements  (more  than  JWARS  support), 
Action:  ASD(C3I)/DSC/Joint  Staff/IC 

—  Populate  the  data  bases,  Action:  ASD(C3I)  DSC/JDS/DD(R&E) 

DMSO 

—  Make  available  to  community.  Action:  JDS 

•  Insight:  C4ISR  analysis  will  require  a  range  of  different  models,  which  may 
include  but  will  not  be  limited  to  JWARS. 

•  Recommendation:  Develop  a  requirement  and  funding  plan  to  develop  the 
analysis  infrastructure  (data,  tools,  etc)  needed  to  respond  to  current 
demands.  This  is  more  than  JWARS  and  JDS.  This  needs  to  be  in  place  by  the 
QDR.  Action:  J-2,  J-6,  J-8,  DSC  (J-6  lead). 
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Analyzing  C4ISR  for  2010 
Smaller  Scale  Contingencies 


Smaller  Scale  Contingencies  (SSC)  proved  difficult  to  define  well  and  difficult  to 
differentiate  from  Operations  Other  Than  War  (OOTW).  The  group  chose  to  focus  its 
efforts  on  mission  types  in  which  the  use  of  military  force  was  likely  (raids,  hostage 
rescue,  opposed  NEOS,  etc.),  which  were  also  discussed  as  those  for  which  military 
considerations  tend  to  take  precedence  over  political  considerations.  This  distinction  was 
adopted  in  order  to  focus  the  Working  Group  on  a  different  arena  from  the  OOTW  group, 
which  focused  most  of  its  energy  on  Peace  Operations  and  Humanitarian  Assistance  and 
Disaster  Relief  missions. 

The  group  recognized  that  this  is  an  artificial  distinction.  Many  formal  definitions  of 
OOTW  explicitly  include  SSC.  Moreover,  SSC  intended  as  nearly  pure  military 
operations  often  are  changed  into  more  political  activities.  The  scenario  played  out  in 
Haiti  (which  went  in  a  matter  of  a  few  minutes  from  an  invasion  of  a  small  country  to  a 
friendly  occupation  with  primarily  political  goals)  provided  a  good  example  of  this 
phenomenon.  Finally,  as  the  group  stressed  in  its  deliberations,  SSC  tend  to  have 
significant  political  dimensions,  even  when  the  operations  themselves  are  largely  military 
in  character. 

The  Working  Group  also  concluded  that  analysis  of  SSC  cannot  be  meaningfully 
conducted  on  the  group  of  missions  as  a  whole.  Their  diversity  requires  that  tools  and 
analytic  efforts  be  focused  on  meaningful,  coherent  subsets  of  the  topic.  For  example, 
studies  of  raids  will  require  different  tools  and  data  than  studies  of  hostage  rescues. 
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Smaller  Scale  Contingencies: 

Participants  List 

NAME 

ORGANIZATION 

Anway- Wiese,  Carol 

Boeing 

Cann,  David 

Naval  Undersea  Warfare  Center, 

Newport 

Chartier,  Chris 

OASD  (C3I)/Decision  Support  Center 

Cherolis,  George 

TRW  Tech  Support  for  TACCSF 

Coe,  Gary 

IDA 

Elton,  Olaf 

MITRE 

Forsythe,  Steven  L.  (Major  USAF) 

AFSAA/SAAB 

Hayes,  Richard  E.  (Chair) 

Evidence  Based  Research  Inc. 

Herlihy,  Anthony  R.  (Major, 

USMC) 

Marine  Security  Guard  Bn. 

Kirkland,  Joseph  K. 

Evidence  Based  Research  Inc. 

Macklin,  Marilyn 

HQ,  Dept,  of  the  Army 

McGregor,  Otis  (Captain,  USA) 

US  Army  LIWA 

Morrell,  Angela 

US  Special  Operations  Command 
SOIO-C4I 

Oswalt,  Ivar 

Kapos  Associates  IncyN6M  Support 

Peterson,  Pat 

SAIC 

Richards,  Dale 

USAF  Research  Laboratory/IFTB 

Siegel,  Adam 

Center  for  Naval  Analyses  (CNA) 

Wheatley,  Gary  (RADM  USN  Ret.) 

Evidence  Based  Research  Inc. 

Who  Participated: 


The  eighteen  individuals  who  participated  are  listed  on  the  slide.  They  represented  an 
excellent  mix  of  C4ISR  analysts  and  operators  with  widely  divergent  backgrounds  and 
training.  While  this  diversity  made  communication  difficult  at  times,  the  members  of  the 
Working  Group  did  an  excellent  job  of  working  with  one  another.  No  one  perspective 
dominated  and  a  remarkably  coherent  and  consistent  perspective  emerged  over  time. 
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Smaller  Scale  Contingencies: 

Characterize  C41SR 

•  Require  more  rapid  &  agile: 

-  Intelligence  processes 

—  Continuous/dynamic  planning  &  execution 

•  Pressure  toward  centralized  control 

—  “Only  game  in  town” 

—  Strategic  implications  of  tactical  activity 

•  Danger  of  different  understandings  of  the  situation  across  command  echelons 


Characterize  C4ISR  (1): 

While  the  Working  Group  was  cognizant  of  the  need  to  differentiate  SSC  from  other  types 
of  operations,  many  of  the  ways  that  C4ISR  needs  are  characterized  in  this  arena  would  be 
the  same  as  those  in  other  arenas.  For  example,  looking  at  future  SSC,  the  dominant 
requirement  will  be  for  more  rapid  and  agile  intelligence,  planning  and  execution  processes. 
While  not  unique  to  SSC,  this  need  will  be  important  and  analysts  must  be  looking  for  ways 
to  ensure  it  is  met.  This  means  exploring  ways  to  improve  the  speed  and  responsiveness  of 
the  entire  ISR  process  (from  tasking  through  dissemination)  as  well  as  to  couple  situation 
awareness,  decision  making,  and  execution  more  closely. 

SSC  are  different  from  MTW,  at  least,  in  that  they  almost  always  involve  considerable 
pressure  for  centralized  control.  This  arises  both  from  the  legitimate  concerns  of  higher 
command  arising  from  the  potentially  strategic  implications  of  military  activities  down  to 
the 

tactical  level  and  the  more  idiosyncratic  fact  that  SSC  are  often  "the  only  game  in  town," 
which  means  that  decision  makers  at  all  levels  are  free  to  invest  time  in  them.  MTW,  on 
the  other  hand,  are  often  large  and  complex  enough  that  senior  political  and  military  officials 
often  are  to  busy  to  focus  on  tactical  level  decisions  and  actions. 

Given  the  high  level  of  attention  likely  to  be  generated,  the  danger  also  arises  of  different 
understandings  of  the  military  and  political  situations  at  different  levels  of  command.  These 
disconnects  can  threaten  the  coherence  and  ultimate  success  of  the  operation. 
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SSC  may  (but  do  not  always)  also  require  a  high  level  of  tactical  security  (particularly  in  the 
planning,  preparation  and  entry  phases)  and  management  of  highly  visible  consequences. 

The  experience  of  conducting  an  operation  that  is  a  military  success  yet  fails  in  terms  of  US 
policy  was  all  too  familiar  to  the  Working  Group  participants. 

Communication  issues  arose  several  times  during  the  discussion.  SSC  often  require  new 
or  novel  infrastructures  that  overcome  gaps  arising  from  military  service  differences,  the 
need  to  include  coalition  partners  or  host  nations,  or  the  need  to  deal  with  non-military 
actors  (inter-agency  or  NGO).  Moreover,  secure  communications  may  be  needed  to  achieve 
adequate  operational  security. 
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Smaller  Scale  Contingencies: 

Characterize  C4ISR  (conCd) 

•  May  Require: 

-  High  level  of  tactical  security 
—  Management  of  highly  visible  consequences 
—  Specialized  communication  capabilities: 

•  Infrastructure 

•  Security 

•  Coalition 
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Smaller  Scale  Contingencies: 

Key  Issues  and  Questions 

•  SSC  not  necessarily  a  lesser  included  case  of  MTW 

—  MOFE  and  MOPE  not  “controlled”  by  the  military 

—  Intelligence  required  tend  to  be  more  fine  grained  and  detailed  than  for 
MTW 

•  Definition  of  mission  success/failure  often  includes  constraints  not  present  in 
MTW. 

•  Tension  between: 

—  Avoiding  mission  failure 
—  Maximizing  likelihood  of  mission  success 


Key  Issues  and  Questions: 

The  Working  Group  wanted  to  emphasize  that  SSC  are  not  necessarily  a  lesser-included 
case  of  MTW .  This  is  crucial  when  data  is  being  selected,  models  chosen,  parameters 
estimated,  or  research  is  being  organized.  Some  of  the  distinctions  were  mentioned 
earlier.  Others  include  the  fact  that  the  MOFE  and  MOPE  are  usually  not  under  the 
control  of  the  military.  In  this  sense,  SSC  are  properly  understood  to  be  OOTW. 
Moreover,  the  group,  and  particularly  the  operators  in  the  group,  stressed  the  fact  that  the 
intelligence  required  in  many  SSC  is  more  fine  grained  and  detailed  than  that  required  for 
the  same  levels  of  command  in  MTW. 

SSC  are  often  conducted  under  restraints  that  would  not  be  present  in  MTW.  Rules  of 
Engagement  (ROE)  and  political  restrictions  often  limit  the  activities  available  for 
mission  accomplishment.  Moreover,  SSC  tend  to  be  characterized  by  a  very  meaningful 
tension  between  efforts  to  avoid  mission  failure  and  maximizing  the  likelihood  of  mission 
success.  Examples  of  decisions  to  limit  the  size  and  armament  involved  in  operations  in 
order  to  avoid  undesirable  political  consequences  were  offered  by  several  members  of  the 
Working  Group. 
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Smaller  Scale  Contingencies: 

Relative  Worth 


Improved  C41SR  in  SSC  should  yield  benefits  such  as: 
—  Economy  of  force 
—  Agility  and  synchronization 
—  Improved  operational  performance 
—  Improved  policy  effectiveness 
—  Improved  mission  accomplishment 


Relative  Worth: 


The  group  noted  that  improved  C4ISR  should  pay  off  in  some  very  specific  ways  in  SSC. 
In  particular,  greater  agility  in  situation  awareness,  planning  and  execution  were  placed 
paramount  in  importance.  Not  only  better  operational  performance,  but  also  improved 
likelihood  of  mission  accomplishment  and  policy  effectiveness  were  considered  possible. 
Additional  benefits  should  include  improved  agility  and  synchronization  and  greater 
economy  of  force. 
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Smaller  Scale  Contingencies: 

Assessing  Relative  Worth 

•  Assessment  of  the  relative  worth  of  C4ISR  systems  requires  considering: 

—  Range  of  SSC  missions  and  their  relative  likelihood 
-  Impact  on  MOFE 

•  Tools  and  methods  for  assessing  relative  worth  require  substantial 
verification,  validation,  and  accreditation  (W&A) 


Assessing  Relative  Worth: 

The  method  for  assessing  the  relative  worth  of  C4ISR  systems  was  seen  as  readily 
available  from  fundamental  operations  research  techniques.  This  is  not  to  say  that  the 
data  and  tools  needed  to  conduct  those  analyses  are  available.  Simply  stated  the  group 
recognized  the  value  of  a  particular  C4ISR  innovation  could  be  seen  as  a  function  of  the 
range  of  SSC  missions  to  which  it  applies,  the  relative  likelihood  of  those  mission  types 
and  their  estimated  impact  on  the  relevant  MOI^  and  MOPE.  After  explicit  discussion, 
the  group  rejected  the  idea  that  the  "importance"  of  the  mission  type  be  cidded  to  the 
equation,  arguing  that  any  one  type  of  mission  might  be  crucial  to  US  national  interests 
under  selected  circumstances.  The  obvious  example  of  this  is  the  crucial  role  of  the 
blockade  during  the  Cuban  Missile  Crisis. 

The  working  group  also  felt  compelled  to  point  out  the  need  for  verification,  validation 
and  accreditation  (VV&A)  for  tools,  methods,  and  data  being  used  to  conduct  these 
relative  worth  analyses. 
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The  group  readily  accepted  both  the  set  of  measures  of  merit  categories  defined  by 
MORS  in  the  1980s  and  the  suggested  change  offered  by  Dr.  Stuart  Starr  during  his 
plenary  address,  of  adding  an  outside  layer,  Measures  of  Policy  Effectiveness  (MOPE). 
Equally  important,  the  group  also  wanted  to  ensure  that  analyses  of  C4ISR  in  SSC  "be 
informed  by"  metrics  from  all  the  relevant  layers.  The  conclusion  was  not  that  all 
analyses  include  all  levels  of  measurement.  For  example,  many  analyses  are  focused  on 
existing  equipment  for  which  the  dimensional  parruneters  are  fixed.  However,  the 
Working  Group  wanted  to  emphasize  the  need  for  analysts  to  be  aware  of  the  limits  of 
analyses  that  do  not  consider  all  the  levels. 


B-9 


Smaller  Scale  Contingencies: 

Measures  of  Merit  (cont*d) 

•  Analysis  of  future  C4ISR  systems  should  be  informed  by  measures  of  merit  at 
all  the  MORS-defined  levels 

•  Interoperability 

•  Distribution  of  information: 

—  Include  military,  interagency,  and  coalition  partners 
—  Appropriate  balance  of  push/pull 


Measures  of  Merit  (1  and  2,  cont’d): 

Interoperability  issues  also  arose  several  times.  Despite  the  emphasis  on  this  topic  since 
Grenada,  it  continues  to  bedevil  operators  in  SSC.  This  also  confounds  (but  is  not  the 
only  confounding  factor)  the  distribution  of  information.  Despite  changes  in  policy 
arising  from  the  Bosnia  experience,  the  people  on  the  spot  who  do  the  planning  and 
support  the  execution  of  SSC  are  often  unaware  of,  or  are  unable  to  acquire  ISR  products 
in  a  timely  manner.  This  problem  arises  across  service,  national  and  functional  lines.  The 
correct  balance  of  information  push  and  information  pull  has  not  yet  been  achieved. 
Metrics  for  interoperability  and  for  information  dissemination  remains  important  in 
assessing  future  C4ISR  systems. 
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Smaller  Scale  Contingencies: 

Measures  of  Merit  (confd) 

•  Quality  of  information  is  the  key  dimension 

-  Responsive  to  war  fighters 

• 

Cover  CCIR 

• 

In  useable  format 

— 

Characterized  as: 

• 

Complete 

• 

Current 

• 

Correct 

• 

Precise  enough  for  purposes 

• 

Consistent:  internally  and  across  command  centers 

Measures  of  Merit  (3): 


The  quality  of  the  information  available  was  seen  as  the  single  most  important  metric  for 
SSC.  This  was  defined  to  include  its  responsiveness  to  warfighters,  including  the 
Commander's  Critical  Information  Requirements  (CCIR),  but  also  the  needs  of  key  staff 
members  and  the  imits  executing  the  operation.  Responsiveness  was  seen  as  including 
much  more  than  mere  availability.  Some  operators  reported  "drinking  from  a  fire  hose" 
when  they  were  given  "all  the  relevant"  material.  Others  noted  that  they  often  received 
material  in  a  format  that  either  was  not  readable  or  was  difficult  to  use.  Here,  again,  the 
proper  balance  of  push  and  pull  is  crucial. 

What  is  meant  by  high  quality  information  was  based  on  the  Headquarters  Effectiveness 
Assessment  Tool,  which  was  familiar  to  several  members  of  the  Working  Group. 
"Perfect"  information  in  a  military  situation  is  seen  as  complete  (for  the  command's 
needs),  current,  correct,  precise  enough  for  the  associated  military  purposes  (which  will 
vary  from  targeting,  which  is  very  precise,  to  general  maneuver  planning,  which  is  often 
much  less  precise)  and  consistent.  Consistent  here  means  both  internally  consistent 
within  the  data,  information,  and  knowledge  of  the  C4ISR  system  and  consistent  across 
command  nodes. 
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Smaller  Scale  Contingencies: 

Measures  of  Merit  (canted) 

*  Anticipation  of  Key  Demands  for; 

—  Intelligence 
—  Logistics 
—  Operations 

-  Communications 

•  Rapid  and  robust  situation  assessment,  decision  making,  planning,  execution: 

-  Speed 

—  Variety  of  alternatives 
—  Quality  (effectiveness) 


Measures  of  Merit  (4): 

The  group  also  wanted  to  stress  the  need  for  future  C4ISR  systems  that  will  be 
anticipatory  across  the  full  range  of  key  demands  that  arise  in  command  and  control: 
operational,  intelligence,  logistics  and  communications.  In  other  words,  if  continuous 
and  effective  planning  and  execution  systems  are  to  work,  the  system  must  be  thoroughly 
cross-linked  in  very  near  real  time. 

Similarly,  measures  of  merit  that  focus  on  rapid  and  robust  decision  making,  planning 
and  execution  will  be  needed.  This  means  we  need  valid  and  reliable  indicators  of  the 
speed  of  the  C4ISR  processes,  the  variety  of  futures  and  alternative  courses  of  action  they 
can  and  do  consider,  and  the  quality  of  the  decisions  they  make.  Indicators  of  decision 
quality  essentially  must  be  MOPE  and  MOPE,  though  MOE  for  C4ISR,  MOP  for  C4ISR 
systems  and  the  dimensional  parameters  that  drive  them  may  also  be  needed  for 
diagnostic  purposes  in  any  given  analysis. 
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Smaller  Scale  Contingencies: 

Tools  Needed 

•  Improved  support  tools  for  CO  A 

—  Credible  projections  of  alternative  futures 
—  Capacity  for  “what  if?”  analysis 
—  Fast  enough  to  support  mission  planning 
—  Must  be  able  to  pull  from  current  operational  databases 

•  Virtual  C4ISR  rehearsal  capability 

—  Analytic  tool 
—  Operational  tool 


Tools  Needed  (1): 

This  area  was  perhaps  the  least  mature  of  the  areas  discussed  in  the  working  group. 
Tension  existed  between  the  desire  to  identify  tools  for  C4ISR  analysts  and  tools  needed 
by  the  operators.  The  requirements  were  seen  to  be  quite  similar  in  many  cases  and  the 
group  identified  several  tools  that  would  be  of  value  to  both. 

The  most  important  tool  requirement  was  for  credible  support  to  course  of  action  (COA) 
analysis.  This  means  tools  that  generate  projections  of  alternative  futures  that  are 
credible  to  operators  and  well  as  the  capacity  for  sensitivity  or  "what  if  analyses.  These 
tools  must  also  be  fast  enough  to  support  mission  planning  and  have  an  analytic  mode 
that  permits  large  numbers  of  runs  in  a  short  period  of  time.  Moreover,  these  tools  must 
be  designed  to  pull  from  operational  databases,  not  just  notional  or  analytic  ones. 

Perhaps  the  most  novel  idea  arising  for  new  tools  was  a  call  for  a  virtual  C4ISR  rehearsal 
capability  that  would  be  designed  both  as  a  tool  for  analysts  and  for  operators.  Operators 
would  use  it  for  training  and  mission  rehearsal.  Analysts  could  use  it  for  improving  their 
understanding  of  missions,  generating  hypotheses  about  C4ISR  improvements, 
examining  man  in  the  loop  elements,  and  testing  hypotheses. 
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Smaller  Scale  Contingencies: 

Tools  Needed  (confd) 


•  Improve  data  collection  and  control 

•  Identify  and  link  simulated  and  real  data 

•  Future  systems  better  integrated  across: 

—  Functions 
—  Echelons  of  Command 
—  Time 

—  Capability  to  model/ameliorate  the  impact  of  language,  culture  and 
training  differences 


Tools  Needed  (2): 


The  group  heard  a  similar  plea  for  more  effective  ways  to  both  link  and  differentiate  real 
and  simulated  data,  particularly  in  real  headquarters  participating  in  virtual  exercises  and 
training. 


In  general,  tools  are  needed  to  assess  the  degree  to  which  C4ISR  systems  are  linked 
across  functions,  echelons  and  time.  New  information  systems  are  linking  operators  more 
and  more  closely,  but  no  tools  to  measure  these  changes  or  their  impacts  are  currently 
available.  Similarly,  models  that  capture  the  effects  of  "soft  factors"  such  as  differences 
in  language,  culture,  or  training  are  not  available  but  would  be  valuable  in  an  era  that 
stresses  "jointness"  and  coalition  operations. 


Smaller  Scale  Contingencies: 

Tools  Needed  (conTd) 

•  Virtual  C4ISR  rehearsal  capability 

—  Analytic  tool 
—  Operational  tool 

•  Simulation  for  rapid  integration  and  deconfliction  of  communications 

•  Rich  relational  databases  with  improved  linkage  software 

•  Uncertainty  estimator  and  communicator 

•  Ability  to  capture  and  replay  data/voice/imagery  traffic  for  current  and 
future  operations 


Tools  Needed  (3): 

The  number  and  variety  of  communications  "horror  stories"  called  out  by  both  the 
plenary  speakers  and  the  operators  in  the  Working  Group  were  seen  as  indicating  a  need 
for  a  simulation  that  would  focus  on  the  capability  to  rapidly  integrate  and  deconflict  the 
"kluges"  of  systems  often  required  in  SSC. 

Rich  relational  databases,  focused  on  the  SSC  mission  areas  were  also  seen  as  important, 
as  was  development  of  improved  linkage  software  to  speed  searches  and  enhance  their 
utility.  Finally,  systems  that  can  estimate  and  communicate  uncertainty  in  the  context  of 
C4ISR  systems  and  the  information  they  present  were  also  seen  as  important. 

Analysts,  particularly  those  supporting  operational  organizations,  also  felt  a  need  to 
capture  and  replay  information  (data,  voice  traffic,  imagery,  etc.)  for  current  and  future 
operations.  This  would  greatly  facilitate  after  action  analyses  from  both  operations  and 
exercises. 
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Smaller  Scale  Contingencies: 

Examining  Illustrative  C4ISR  Smaller  Scale 

Contingencies 


Mission 

Types 

Future  of 
C4ISR 

Analytic 

Tools 

Measures 
of  Merit 

Key  Metrics 

Raid 

>  3D 
Imagery 

y  Predictive 
Modeling 

>  MOFE> 
MOPE 

> 

Mission  Accom¬ 
plishment 

>  Virtual 
Rehearsal 

>  MOP 

> 

Collateral 

Damage 

> 

Interoperability 

> 

Speed 

> 

Targets 

Destroyed 

> 

Minimum  US 
Casualties 

Illustrative  Examples  (1)  -  Raid: 


Slides  15  through  18  represent  detailed  work  performed  by  the  working  group,  which 
actually  examined  four  different  types  of  SSC  mission  areas  to  link  future  C4ISR  needs, 
the  analytical  tools  required  to  support  their  development  and  refinement,  the  most 
crucial  measures  of  merit,  and  the  implied  key  metrics. 

In  the  Raid  analysis  the  group  determined  that  the  critical  C4ISR  product  is  timely  3D 
Imagery.  This  imagery  is  essential  to  support  the  needed  analjdic  tools  of  Predictive 
Modeling  and  Virtual  Rehearsal,  which  were  in  turn  believed  to  offer  potential  for  greatly 
improved  mission  success.  Measures  of  merit  are  dominated  by  MOPE  and  MOPE,  with 
Force  Effectiveness  deemed  somewhat  more  important  the  Measure  of  Policy 
Effectiveness.  Key  Metrics  include  Minimum  US  Casualties,  Minimum  Collateral 
Damage,  Interoperability  of  C3I  and  number  of  Targets  Destroyed  in  the  raid. 
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Smaller  Scale  Contingencies: 

Examining  Illustrative  C4ISR  Smaller  Scale 
Contingencies  (conCd) 


Mission 

Future  of  C4ISR 

Analytic  Tools 

Measures  of  Merit 

Key 

Types 

Metrics 

Urban 

> 

Global  Positioning 

> 

Predictive 

> 

Inter-operability 

> 

Minimum  US 

Warfare 

System 

Modeling 

Casualties 

> 

Databases  as  Intel 

> 

Virtual 

> 

Economy  of 

> 

Collateral 

Products 

Rehearsal 

Force/Cost 

Effective-ness 

Damage 

> 

Communications 

> 

Maneuver 

> 

Maneuver¬ 

> 

Interoperability 

Continuity 

Models 

ability 

of  communi¬ 
cation  and 

computers 

> 

Identification 

> 

Intel/Ops 

> 

*'Key  Nodes  to 

Friend  or  Foe  for 

Automation 

Take*'  in  urban 

land  forces 

terrain 

> 

Language 

> 

CommunE 

Translators 

cations 

simulation 

Illustrative  £xainples(2)  --  Urban  Warfare: 

Urban  Warfare  offers  perhaps  the  biggest  C4ISR  challenges  today  and  in  the  future. 
These  include  enhanced  utilization  and  exploitation  of  GPS  for  precise  three  dimensional 
positioning.  Combat  Identification  of  friend  and  foe  in  densely  populated  areas  where 
non-combatants  are  often  mixed  with  the  enemy,  and  communications  that  will  function 
seamlessly  from  sewers  to  skyscrapers.  Detailed  databases  are  essential  to  support 
Predictive  Modeling  and  Virtual  Rehearsal  tools.  Measures  of  Merit  stress 
Interoperability  and  Economy  of  Force  with  key  metrics  similar  to  those  in  the  previous 
example. 
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Smaller  Scale  Contingencies: 

Examining  Illustrative  C4ISR  Smaller  Scale 

Contingencies  (confd) 

Mission  Types 

Future  of 

Analytic  Tools 

Measures  of 

Key  Metrics 

C4ISR 

Merit 

Counter- 

>  Relational 

> 

High  volume 

Hits  verses 

>  Decrease 

terrorism/ 

databases 

of  data 

false  alarms 

in 

Operational 

with  linkage 

terrorist 

Disruption 

software 

attacks 

>  Better 

> 

Knowledge- 

>  Level  of 

>  Increase 

organized 

able  domain 

confidence 

in 

data 

experts 

terrorist 

threat 

>  Threshold 

> 

Prioriti- 

>  Level  of 

control 

zation 

^'tagsyor 

intelligence 

alerts 

uncertainty 

Illustrative  Exainples(3)  ~  Counter  Terrorism/Operational  Disruption: 


Terrorism  and  other  trans-national  threats  are  quickly  becoming  epidemic.  As  developed 
in  previous  counter-narcotics  analyses,  terrorists,  drug  czars  and  similar  actors  can  be 
identified  and  located  through  concentrated  research  and  analysis.  The  key  is  the  data, 
and  future  C4ISR  to  address  these  threats  will  require  much  better  organized  data  that  can 
be  linked  through  relational  databases.  Analytic  tools  can  then  use  the  data  to  identify 
and  locate  terrorists.  Measures  of  Merit  will  address  the  success  of  the  analyses  and 
focus  on  accuracy  of  hits  versus  false  alarms,  and  the  confidence  level  of  the  alerts. 
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Smaller  Scale  Contingencies: 

Examining  Illustrative  C41SR  Smaller  Scale  Contingencies 

(cant’d) 

Mission  Types 


Illustrative  £xamples(4)  —  Opposed/Potentially  Opposed  NEO: 

An  Opposed  or  Potentially  Opposed  NEO  is  perhaps  one  of  the  most  difficult  and 
delicate  missions  which  our  armed  forces  are  called  upon  to  conduct.  These  missions 
literally  mean  life  or  death  to  US  and  other  friendly  civilians,  and  create  high  visibility 
political  consequences.  Key  C4ISR  capabilities  needed  are  detailed  and  current 
intelligence  products  to  include  precise  terrain  and  building  data,  and  continuous  or  near 
continuous  location  information  about  evacuees  and  potential  opponents.  M&S  tools  that 
provide  rehearsal  capability  and  COA  analyses  are  particularly  needed.  In  this  mission 
MOPE  and  MOFE  are  paramount  with  MOPE  most  important.  Mission  accomplishment 
with  minimum  US  and  civilian  casualties  are  the  Key  Metrics. 


Opposed  NEO/ 
Potentially 
Opposed  NEO 


Future  of  C4ISR  Analytic  Tools 

>  Information  >  Modeling  & 

on  terrain  Simulation 

Anticipatory 

>  Improved  >  COA 

interagency  Analysis 

/  coalition  Tool 

C4ISR 

>  Ability  to 
monitor 
situation 
and 

operative 

environment 

>  Detailed 
Intel 
products 


Measures  of  Key  Metrics 
Merit 

>  MOFE<  >  Mission 

MOPE  Accomplish¬ 

ment 

>  Quality  of  >  Minimum  US 

informatio  and  civilian 

n  casualties 

>  MOP  >  Increase  in 

capability  of 
number  of 
people 
evacuated 
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Smaller  Scale  Contingencies: 

Assessment  of  State -o  f -the- Practice 


Assessment: 

This  slide  is  self-explanatory  except  that  the  Working  Group  felt  that  many  areas  fell 
between  red  and  yellow  and  chose  to  color  them  orange. 
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Smaller  Scale  Contingencies: 

Recommendations 

•  Develop  clearer  deHnitions  to  support  analysis 

•  Focus  on  speciHc  missions  rather  than  artificial  categories 

•  Baselines  are  essential  for  good  analyses  and  must  be  developed  for  the 
mission  types  included  in  SSC 

•  Invest  in  the  C4ISR  analysis  tools  identified  by  the  working  group  (e.g., 
W&A) 


Recommendations  (1): 

The  recommendations  were  difficult  to  draft  and  represent  only  a  fraction  of  the  rich 
discussion  and  debate  of  the  working  group.  Those  recommendations  ranged  from; 

•  the  difficulty  of  defining  SSC  (ultimately  resolved  in  the  very  practical  way  of 
noting  that  the  included  mission  types  are  really  the  focus  of  analysis) 

•  the  argument  that  baselines  do  not  exist  for  most  of  these  mission  types 

•  meaningful  analysis  must  have  those  cmcial  baselines  for  assessment  of  the 
contribution  of  current  and  future  C4ISR  systems. 

The  most  fundamental  recommendation  was  that  DoD  and  the  MORS  community  begin 
an  investment  program  in  the  tools  identified  as  necessary. 
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The  area  of  MOPEs  was  also  called  out  as  an  area  crucial  to  further  progress  in  SSC. 
This  finding  was  expected  to  be  similar  to  one  reported  by  the  OOTW  Working  Group. 
Investment  in  databases  (and  information  and  knowledge  bases)  to  support  SSC  mission 
area  analyses  was  also  identified  as  a  very  important  recommendation. 
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Smaller  Scale  Contingencies: 

Summary 


•  Future  and  current  needs  of  C4ISR  within  SSC  must  be  mission  focused 

•  SSC  require  more  detailed,  “fine  grained”  data  from  C4ISR  elements 

•  Investment  in  relevant  C4ISR  information  and  C4ISR  COA  tools 
recommended 

•  Development  of  baselines,  information  and  further  research  necessary 


Sununary: 


The  current  and  future  C4ISR  needs  for  Smaller  Scale  Contingencies  must  be  addressed 
within  the  context  of  each  specific  mission  category,  as  opposed  to  addressing  a  broad 
range  of  similar  missions.  By  heeding  this,  proper  analysis  of  each  mission  area  may  be 
secured.  C4ISR  data  within  Smaller  Scale  Contingencies  will  need  to  develop  a  higher 
level  of  tactical  detail  and  control  then  other  arenas  of  warfare  (i.e.  peacekeeping,  peace 
enforcement,  major  theater  war.)  This  characteristic,  developed  from  tight  operational 
constraints,  will  aid  commanders'  tactical  planning,  control  and  security.  Investment  in 
such  C4ISR  information  systems  such  as  virtual  rehearsal  programs,  course  of  action, 
analytical  tools  and  relational  databases  is  recommended  to  increase  military  force 
flexibility  and  performance. 


Analyzing  C4ISR  for  2010 
Operations  Other  Than  War 
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Operations  Other  Than  War: 

Working  Group  Taskings 

•  Characterize  C4ISR  within  WG  focus  area 

—  Decompose  the  elements  of  C4ISR 

•  Deflne  the  relative  worth  of  C4ISR 

—  C4ISR/Firepower/Force  Employment 

•  Develop  &  Recommend  Measures  of  Merit 

—  MOPs  and  MOEs 

-  How  well  they  value  C4ISR,  estimate  ROI 

•  Identify  and  describe  tools 

•  Analyze,  synthesize,  and  infer 


Working  Group  Tasking: 

The  Terms  of  Reference  charged  the  Working  Group  to  undertake  five  tasks: 

1.  Across  the  range  of  OOTW  missions,  decompose  the  aspects  of  C4ISR  and  consider 
the  analysis  issues  associated  with  them.  The  objective  is  to  consider  the  value  of 
CISR  contributions. 

2.  Continue  the  assessment  by  considering  the  relative  worth  of  C4ISR  to  “firepower” 
and  force  employment.  Note  here  that  “firepower”  is  a  misnomer  for  force 
capabilities  and  systems,  since  generally  the  missions  and  tasks  of  OOTW  will 
require  other  than  the  prime  warfare  systems. 

3.  Considering  the  types  of  analysis  to  be  performed,  develop  measures  of  merit  (MOPs 
and  MOEs)  that  are  appropriate  to  the  mission  areas.  Also,  address  the  ability  to 
conduct  assessments  of  return  on  investment. 

4.  Identify  tools  and  methods  that  are  necessary  to  the  analysis  described  above. 

5.  Finally,  consider  all  the  information  generated  to  draw  conclusions  and  observations. 
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Operations  Other  Than  War: 

Group  Participants 

• 

Art  Dougas 

Booz-Allen 

• 

John  Furman 

Mitre 

• 

Bob  Hartling 

OPNAV  81 

• 

John  Kelsey 

TASC 

• 

Randall  (Mel)  Parish 

TRAC-WSMR 

• 

Terry  Prosser 

Logicon 

• 

Mark  Sinclair 

Joint  C4ISR  Battle  Center  (Veridian) 

• 

Mike  Sovereign 

Naval  Postgraduate  School 

• 

Cy  Staniec 

Logicon 

• 

Don  Theune 

Virtual  Technology  Corporation 

• 

Corinne  Wallshein 

AFSAA/SAAB 

• 

Scott  Welch 

Evidence  Based  Research 

• 

Russ  Richards 

Mitre  (Synthesis  Group) 

OOTW  Group  Participants: 


Dr  Cy  Staniec  chaired  the  group,  with  John  Furman  and  Terry  Prosser  as  co-chairs.  The 
group  included  a  broad  array  of  skills  and  experiences,  including  recent  experience  in 
PACOM  activities  (Prof  Mike  Sovereign  and  Mark  Sinclair)  and  Bosnia  (Terry  Prosser). 
Dr  Russ  Richards  participated  as  a  member  of  the  Synthesis  Working  Group. 


C-3 


Operations  Other  Than  War: 

Background 

•  1996-  PACOM  Workshops 

•  January  1997  -  MORS  OOTW  Workshop 

•  October  1997  -  HA/DR  IN  THE  NEXT  CENTURY 

-  CCRP  report 

-  VIC  Concept  description 

•  April  1998  briefing  “Peacekeeping  in  Bosnia-  New  Frameworks  for 
Analysis” 


Background: 

Previous  workshops  and  activities  provided  groundwork  for  this  working  group.  Several  of  the 
participants  had  been  involved  in  the  PACOM-sponsored  workshops  addressing  OOTW  analysis 
held  in  1996.  In  January  1997,  MORS  also  conducted  a  workshop  on  analysis  methods  and  tools 
for  OOTW.  The  C4ISR  group  characterized  the  complexities  posed  by  the  OOTW  missions,  and 
cited  some  of  the  differing  analysis  requirements  and  the  need  for  tools  to  support  analysis. 

Further  background  work  was  provided  via  a  report  entitled  Humanitarian  Assistanrp;  anH 
Djs^ter  Relief  in  the  Next  Century.  (DoD  C4ISR  Cooperative  Research  Program,  October  28-30 
1997).  This  report,  written  by  Professor  Mike  Sovereign,  documented  working  groups  focused  in 
three  main  areas:  collaboration,  cooperation,  and  coordination  (C3)  for  HA/DR;  information 
acquisition  for  HiVDR;  and  crisis  management  in  HA/DR  operations.  Mark  Sinclair  gave  an 
overview  of  the  Virtual  Information  Center  concept  conceived  in  the  workshop  and  demonstrated 
in  PACOM.  The  VIC  concept  proved  valuable  to  support  crisis  planning  and  allowed  innovative 
analysis  and  problem  solving  in  these  missions. 

Finally,  Ter^  Prosser  presented  a  briefing  entitled  “Peacekeeping  in  Bosnia  -  New  Frameworks 
for  Analysis”  that  reflected  the  issues  and  measures  he  was  faced  with  while  serving  as  analyst  on 
the  Bosnia  task  force  in  1997. 

These  inputs  provided  a  focused  starting  point  for  the  efforts  of  this  working  group. 
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In  order  to  address  its  tissigned  tasks,  the  group  worked  through  a  decomposition  process 
depicted  on  this  slide.  For  the  set  of  OOTW  missions,  we  first  derived  a  set  of  typical 
functions  within  the  missions  and  identified  typical  mission  scenarios  for  analysis.  For 
the  scenarios,  we  considered  how  C4ISR  means  were  employed  in  the  mission.  By 
crossing  the  C4ISR  means  with  the  mission  functions,  we  were  able  to  identify  the  most 
interesting  C4ISR  evaluation  issues.  Finally,  these  led  to  a  categorization  of  typical 
measures  of  merit  and  the  appropriate  tools  to  assess  them.  The  results  of  the  process  are 
presented  in  the  following  slides. 
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Operations  Other  Than  War: 

Functional  Characteristics  of  OOTW 


PK 

PE 

PB 

H  A/DR 

CS 

N  A 

warn 

Security  Opns 

X 

X 

X 

X 

Police  O pns 

X 

X 

X 

X 

X 

X 

Refugee  Opns 

X 

X 

X 

X 

R  esettlem  en  t 

X 

X 

X 

X 

C  om  pliance 

X 

X 

X 

Inspection 

X 

X 

X 

X 

X 

X 

X 

X 

X 

■  iicki  j  j  j  IWfffW 

X 

X 

X 

C  ivil  A  ffairs 

X 

X 

X 

X 

Sustenance 

X 

X 

X 

X 

M  edical  Opns 

X 

X 

X 

X 

G  overnance 

X 

X 

X 

X 

Key: 

PK:  Peacekeeping 

PE:  Peace  Enforcem ent 

PB;  (Peace  Builiding) 

HA /DR  Humanitarian  A  ssistance/D  isaster  R  clief 

CS:  Civil  Support 

NA;  N aiion  a ssistance 

CD;  Counter-drug 

Cl:  Counterinsurgency  (support) 


Mission/Functional  Characteristic  Matrix: 

The  decomposition  step  identified  the  twelve  major  functions  associated  with  OOTW 
missions.  The  missions  considered  were  peace  keeping,  peace  enforcement  and  peace 
building  (PK/PE/PB),  humanitarian  assistance  and  disaster  relief  (HA/DR),  civil  support, 
nation  assistance,  counter  drug  and  counterinsurgency.  By  agreement,  any  other  mission 
in  which  combat  might  be  anticipated  was  left  to  the  Small  Scale  Contingency  working 
group. 

From  this  decomposition,  it  is  easy  to  assess  where  C4ISR  means  may  be  significant  to 
mission  accomplishment  for  each  mission.  Because  of  the  limited  time  to  the  workshop 
and  the  particular  expertise  of  the  group,  we  focused  on  the  PK/PE/PB  and  HA/DR 
mission  areas. 
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Operations  Other  Than  War: 

OOTW  Mission  Characteristics 

•  Characteristics  of  PK/PE 

•  Characteristics  of  HA/DR 

—  Focus  Different : 

—  Focus  Different: 

•  Prevent  conflict 

•  Stop  the  dying 

•  Restore  governance 

-  Variety  of  participants 

*  Stabilize  Situation 

(players) 

•  Exit 

—  Operations  last  beyond  US 

Military  participation 

—  Multiple  participants 

-  Small  footprint 

—  Operations  last  beyond  US 

-  Definition  of  winning  is 

Military  participation 

different  (“hearts  and 

—  Security  operations  are 

minds”) 

Primary 

—  Level  of  force  varies 

Different  Missions -  Different 

ihilar  Problems  to  Address 

OOTW  Mission  Characteristics: 


In  order  to  get  to  the  mecisures  of  merit  necessary  for  analysis,  two  subgroups  were  formed. 

One  group  considered  the  Humanitarian  Assistance/Disaster  Relief  mission  area  and  the 
other  considered  Peacekeeping  and  Peace  Enforcement.  In  each  case,  the  subgroup  listed 
some  of  the  mission  objectives  and  characteristics  that  would  drive  analysis  objectives. 

In  each  case,  it  was  clear  that  the  main  mission  objectives  were  quite  different  than  the 
mission  objectives  of  mid-intensity  warfare.  In  the  case  of  PK/PE,  mission  objectives 
focus  on  security  operations,  prevention  of  conflict,  and  rebuilding  of  governance  and 
economic  stability.  On  the  other  hand,  the  focus  of  HA/DR  is  on  emergency  support, 
including  objectives  like  "stop  the  dying.” 

In  both  czises,  missions  involve  multiple  participants,  usually  beyond  the  US  military.  In  many 
cases,  a  small  footprint  in  desirable,  though  expeditious  completion  of  the  mission  is  also  desired. 
And  the  forces  necessary  for  mission  completion  will  be  tailored  mix.  In  the  case  of  PK/PE, 
combat  and  combat  support  forces  capable  of  preventing  combat  and  providing  security  are 
important.  In  HA/DR,  engineers,  medical  support  are  appropriate. 

These  observations  mean  that  the  measures  used  in  analyzing  and  assessing  these  missions  will 
both  differ  greatly  across  the  missions  and  will  also  differ  greatly  from  the  type  of  measures  used 
in  theater  warfare.  This  will  be  demonstrated  in  greater  detail  in  the  next  slides.  As  might 
be  expected,  analysis  of  C4ISR  within  these  missions  will  pose  a  variety  of  challenges 
substantially  different  than  those  faced  in  theater  warfare. 
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Operations  Other  Than  War: 

Example  Decomposition  -  PE/PK 

CLASS 

MOE 

DATA 

SOURCES 

SYSTEMS 

Mission 

Accom¬ 

plishment 

Freedom  of  Movement 

DeHne  boundaries,  # 
crossings 

Observations  & 
Cameras 

HUMINT 

SIGINT 

Civil  Affairs 

Police  H/R  violations 

Define  &  Count 

Media  &  Reports 

Public 

Compliance  violations 

Define  &  Count 

Inspections  & 
Reports 

Affairs 

PSYOPS 

Polling  results 

Conduct  polls,  collate 

PSYOPS  & 

Public  Affairs 

IMBVT 

Maneuver 

Government  Sovereignty 

Define,  Count  &  Monitor 

10  &  Reports 

Military 

Police 

Media  trends 

Collect,  review,  &  collate 

Civil  Affairs  & 
Public  Affairs 

Signal  Corps 
INTEL 

Economic  trends 

Market  Analysis 

International 
Organizations  & 
Counters 

Weather 

personnel 

Air  Traffic 

Health  trends 

Disease  Rates 

lO  &  indigenous 
reports 

Control 

Medical 

Freedom  of  Action 

Cost 

Time  to  complete 

Force 

Protection 

ThreatCon 

Monitor 

Intel  /  Sensors  / 
Reports 

Casualty  trends 

Monitor  /  Report 

Forces  / 

Reachback 

Terrorist  Incidents 

Monitor  /  Report 

Forces /Intel 

Credibility 

Monitor  /  Report 

Forces /Intel 

Example  Decomposition  —  PK/PE: 


These  two  slides  show  decomposition  for  the  PK/PE  mission  area.  This  particular 
decomposition  is  based  on  experience  with  the  Bosnian  situation,  but  it  is  worth  noting  that  the 
particular  result  would  change  for  different  situation. 

General  areas  of  interest  are  in  mission  completion  measures,  force  protection  measures, 
decision  making  measures,  and  interoperability  measures.  Within  these  areas,  the 
decomposition  addresses  measures  that  are  appropriate  for  C4ISR  tradeoffs. 
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Operations  Other  Than  War: 

Example  Decomposition  (confd)  -  PE/PK 

CLASS 

MOE 

DATA 

SOURCES 

Decision  Making 

Difference  between  perception  & 
truth 

Uncertainty  /  Risk 
Mitigation 

Historical 

Assessment 

Right  info  ? 

Trend  &  Pattern 
analysis 

Timely  info  ? 

Reaction  time} 

<  Planning 

Decision  time} 
Horizon 

Interoperability 

#  Interpreters  required 

Number 

Network  Connectivity 

Systems,  Networks 

Communications  Architecture 
Complexity 

Bandwidth 

Unity  of  Effort 

Compliance  with 
orders 

COP 

Responsiveness 

Response  Time 

Example  Decomposition  —  PK/PE  (cont’d): 


The  mission  completion  category  includes  measures  that  address  stability  in  the  area  of 
operations  at  the  “measure  of  force  effectiveness”  level.  The  sort  of  measures  developed  (e.g. 
freedom  of  movement,  economic  and  health  trends)  are  diverse  and  may  involve  C4ISR  in  their 
completion.  Therefore,  C4ISR  trades  are  viable  and  indeed  important.  Notably,  they  are 
challenging  to  analyze  because  tools  to  address  these  measures  are  scant.  In  the  theater,  current 
experience  is  to  assess  progress  by  trend  analysis  of  data  actually  gathered.  In  planning  cases, 
tradeoff  analyses  could  be  conducted  by  comparing  potential  effectiveness  and  cost  of  using 
alternative  collection  means.  Trades  based  on  impact  on  economic  or  cultural  response  in 
theater  would  currently  be  nearly  impossible  to  assess. 

Force  protection  measures  lend  themselves  to  tradeoff  analysis  since  they  can  be  addressed  as 
effectiveness  and  cost  issues.  On  the  other  hand,  trades  are  more  difficult  to  assess  in  the 
context  of  decision  making  support,  since  the  scenarios  and  models  are  not  widely  available. 
Perception-based  models  are  lacking  in  OOTW,  just  as  they  continue  to  be  lacking  for  war 
fighting  missions.  In  the  area  of  interoperability,  several  measures  can  be  addressed 
analytically  since  they  are  more  closely  related  to  network  performance. 

The  sort  of  decomposition  presented  here  illustrates  some  of  the  needs  and  possibilities  for 
C4ISR  tradeoff  analysis.  This  example  will  apply  for  many  situations,  but  is  certainly  not 
complete.  Further  work  to  extend  these  results  is  warranted. 
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Operations  Other  Than  War: 

Measures  of  Merit 

•  PK/PE 

•  HA/DR 

Internal  (among  C4ISR  ^sterns)  — 

Internal  (among  C4ISR  ^sterns) 

• 

Information  Timeliness 

• 

Timeliness 

• 

Info  Completeness/ 

• 

Completeness/Accurac^ 

Accuracy 

• 

Footprint 

• 

Network  Completeness 

• 

Risk/Flexibility 

• 

Footprint  ~ 

Risk/Flexibility 

External  (between  C4ISR  and 
others,  or  between  C4  and  ISR) 

• 

Mission  Impact  /  $$ 

• 

Architecture  Complexity/ 

Interoperability 

• 

Efficiency  of  Operation 

• 

Common  Operational 

• 

Perception  of  Operation 

Picture 

Reduction  of 
Mortality/Morbidity 

—  External  (between  C41SR  and  others^ 
or  between  C4  and  ISR) 

• 

Non-surprise/Prevention 

• 

• 

• 

• 

• 

Mission  Impact  /  $$ 

Efficiency  of  Operation 

Perception  of  Operation 

Political  Economic  Impacts 

Conflict  incidents 

Measures  of  Merit: 


An  overview  of  measures  for  PK/PE  and  HA/DR  as  applied  to  tradeoffs  within  C4ISR, 
and  between  C4ISR  and  other  forces  and  systems  is  presented  here.  For  trades  within 
C4ISR,  the  measures  frequently  can  be  tied  to  measures  of  network  performance.  When 
this  is  true,  typical  analytic  capabilities  can  be  brought  to  bear. 

When  trades  are  to  be  conducted  outside  C4ISR,  the  measures  are  more  MOFE-level. 
They  will  tend  to  differ  for  each  of  the  mission  areas  in  OOTW.  they  also  are  not 
supported  by  well-developed  tools  and  models  to  conduct  the  analysis.  Mission 
efficiency,  in  particular,  will  be  related  to  economic  and  cultural  measures.  The  theory 
and  transfer  functions”  from  C4ISR  effectiveness  (or  any  other  system  or  force 
capability)  are  lacking  in  most  areas. 
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Operations  Other  Than  War: 

Tools 


Analytic  Purpose 


Situation  Assessment 


Identify  trends  and  patterns 

Correlate  across  classes  of  MOEs 

Political/economic/military 

Ethnic/cultural/religious 

Infer  cause  and  e^ect 

Track  progress  towards  objectives 


Support  to  Decision  Makineand  C2 


Analyze  alternative  courses  of  action 
Identify  requirements 
Reduce  uncertainty 
Improve  timeliness 

Assess  quality  of  interoperability  and  comms 


Trade-Off  Analvsis 


•  Support  QDR-Iike  force  trades 

*  Understand  Total  Force  implications  of  OOTW 


Analytic  Tools 


Polling 

Statistical  Analysis 
Sampling 
Hypothesis  Testing 
Correlation 
Data  display 

Data  from  past  OOTWs/data  mining  tools 
OOTWIPB 


Knowledge-based  expert  systems 
Panels  of  experts 
Virtual  Information  Centers 
Wargaming 

Collaborative  decision  making  tools 
Network  analysis  tools 
Functional  capability  models 
Lessons  learned  databases 


Analytic  frameworks 
Scenarios 

Data  from  past  OOTWs 
Expert  judgment 
Predictive  capability  models 
OPTEMPO  models 


Tools: 


This  slide  portrays  the  types  of  tools  that  are  needed  to  analyze  OOTWs.  It  is  organized  in 
terms  of  major  analytic  purposes.  The  first  two  purposes  (situation  assessment  and  support 
to  decision  making  and  C2)  are  inherent  in  the  planning  and  conduct  of  OOTWs.  Trade¬ 
off  analysis  addresses  the  broader  issue  that  the  workshop  was  charged  to  address:  how  to 
trade-off  C4ISR  against  other  categories  of  things,  such  as  forces,  to  support  QDR-level 
force  trades. 
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Tools  (cont’d): 


While  the  tools  that  support  these  different  purposes  are  organized  into  separate  "bins"  on 
the  slide,  in  fact  there  are  broad  analytic  needs  that  cross  these  categories.  These  include: 

•  Better  ways  to  gather,  analyze,  and  display  data 

•  An  understanding  of  past  OOTWs  and  lessons  learned 

•  Ways  of  tapping  expert  judgment.  A  promising  example  is  the  Virtual  Information 
Center  (VIC)  concept  for  HA/DR  being  developed  for  PACOM 

•  Analytic  frameworks  for  OOTW 

While  an  end-to-end  modeling  approach  to  OOTW  is  well  beyond  the  present  state  of  the 
art,  there  is  a  clear  need  for  tools  and  models  that  will  do  part  of  the  job  within  the  context 
of  an  analytic  framework.  For  example,  tools  for  HA/DRs  that  help  to  predict  damage  and 
casualties  and  generate  requirements  for  medical  supplies  and  personnel  are  invaluable. 
Within  peace  operations,  models  for  assessing  the  performance  of  difference  ISR 
capabilities  or  communications  networks  would  be  similarly  helpful. 
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Operations  Other  Than  War: 

Relative  Worth/Trade  Offs 


•  Trade-off  taxonomy 

—  Internal  -  within  C3  and  ISR 
—  Internal  -  between  C3  and  ISR 

—  External  -  between  C4ISR  and  everything  else  (e.g.,  forces) 

•  Trade-offs  are  progressively  more  difficult  from  internal  to  external 

—  MoMs  go  from  relatively  technical  and  quantifiable  to 
operational/political  and  qualitative 

—  Cause  and  effect  relationships  supporting  higher-level  MoMs  become 
problematic 


Relative  worth/Trade-offs: 

Trade-offs  within  the  categories  of  ISR  and  C3  can  be  done  in  traditional  ways  that  apply 
to  other  military  operations.  Measures  of  perform<ince  and  effectiveness  in  these  areas 
(such  as  quality,  quantity,  and  timeliness  for  ISR)  are  well  understood  and  relatively 
quantifiable.  When  one  tries  to  trade-off  ISR  for  C3  or,  more  broadly,  C4ISR  for  other 
things  such  as  maneuver  forces,  the  capabilities  are  unlike,  and  causal  relationships  leading 
to  the  achievement  of  measures  of  force  effectiveness  or  measures  of  policy  effectiveness 
are  highly  uncertain.  For  this  reason,  the  trade-offs  that  commanders  make  during  every 
OOTW  tend  to  be  qualitative  and  informed  by  expert  judgment  rather  than  quantitative 
analysis.  Qualitative  trade-off  judgments  can  be  just  as  good  as  quantitative  ones.  In  fact, 
in  this  area,  they  may  be  better  than  judgments  based  on  trying  to  force  the  numbers  to  fit 
the  problem.  However,  the  ability  to  make  qualitative  trades,  whether  for  operational  or 
force  planning,  can  be  significantly  improved  by  enhanced  capabilities  in  the  areas 
indicated  by  the  slide: 

•  Definition  of  missions  and  requirements 

•  Exploitation  of  past  lessons  learned 

•  Databases  of  operational  data 

•  Frameworks  for  analysis  with  tools  and  models  integrated  as  appropriate 
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Operations  Other  Than  War: 

Relative  Worth/Trade  Offs  (cant’d) 


•  Easier  trades  (e.g.,  vdthin  ISR)  may  be  helpful  for  programmatic  and  mission 
planning  issues  but  are  less  likely  to  shed  light  on  QDR-level  force  trades 

•  Trades  of  C4ISR  with  forces  for  OOTW  require: 

—  Better  deHnition  of  missions  and  requirements 
—  More  organized  exploitation  of  past  lessons  learned 
—  Databases  of  operational  data 

-  Frameworks  for  analysis  and  appropriate  supporting  tools 
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Operations  Other  Than  War: 

Assessment  of  Current  State-of-the-Practice 


Assessment  of  Current  State  of  the  Practice: 

It  is  not  surprising  to  see  that  the  state  of  the  practice  assessment  shows  many  areas  of 

shortfall. 

•  Structuring  of  the  Problem  rated  a  Yellow/Red.  Recalling  the  focus  of  conducting 
analysis  and  tradeoffs  of  C4ISR  assets,  we  see  that  there  are  few  developed  tools  and 
procedures  to  apply  in  the  general  area  of  OOTW.  Nevertheless,  we  acknowledge 
that  these  missions  are  being  continually  addressed  day-to-day  with  apparent  success. 

•  Human  factors  and  organization  rate  a  red  because  the  human  elements  of  C4ISR  are 
not  well  modeled  in  OOTW.  The  human  performance  aspects  of  C4ISR  performance 
metrics  are  less  difficult  than  the  responses  of  the  actors  in  the  OOTW  area  of 
operations.  That  is,  the  connectivity  from  C4ISR  actions  to  cultural,  religious,  or  other 
reactions  is  lacking,  limiting  the  ability  to  conduct  MOE-based  analysis. 

•  Scenarios  rated  a  yellow/green.  The  reason  is  simply  that  recent  history  has  provided 

a  large  number  of  plausible  planning  scenarios.  During  the  last  QDR,  a  body  of 
approximately  50  scenarios  was  developed  for  the  Dynamic  Commitment  wargame 
series.  The  yellow  results  because  the  scenarios  are  not  well  documented  and  available 
for  planning  use  at  this  time. 
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Assessment  of  Current  State  of  the  Practice  (cont’d): 

•  Measures  of  Merit  is  rated  as  red,  tending  toward  yellow.  The  workshop  efforts  as  well 
as  the  operational  missions  over  the  last  few  years  have  now  organized  this  area  of 
investigation.  Progress  is  being  made  in  identifying  these  measures.  Although  the 
diversity  of  the  OOTW  missions  makes  this  more  difficult,  trends  are  starting  to  emerge 
regarding  what  types  are  appropriate. 

•  Tools  and  applications  rate  red,  trending  toward  yellow.  In  general,  these  missions  are 
diverse  and  complex  enough  that  no  monolithic  model  is  viewed  as  feasible  or  desirable. 
Rather,  what  is  desired  are  sets  of  tools  that  address  parts  of  the  analyses  directly  and  can 
be  intelligently  composed  to  address  more  complex  questions.  A  few  mission  and  force 
building  tools  are  available  and  some  C4ISR  issues  can  be  undertaken  with  standard 
network  tools.  However,  the  connection  from  C4ISR  performance  to  mission  effectiveness 
is  missing,  as  was  indicated  under  structuring  the  problem. 

•  Data  is  considered  red.  Evidence  is  found  in  the  difficulty  to  access  any  data  associated 
with  any  of  the  recent  OOTW  missions  the  US  has  been  involved  with.  Furthermore, 
operational  planning  continues  to  be  hampered  by  difficulty  in  rapidly  accessing  useful 
data.  Due  to  the  difficulties  with  measures,  tools  and  data,  it  follows  that  areas  like  risk 
assessment  would  also  rate  red.  Indeed,  finding  the  marginal  benefit  or  the  risk  associated 
with  any  analysis  topic  tends  to  be  a  difficult  task. 

•  Finally,  reporting  rates  a  red  trending  toward  yellow.  Until  recently,  OOTW  reports  were 
hard  to  come  by.  During  the  QDR,  efforts  to  assess  past  efforts  were  hampered  by  scant, 
inconsistent  data  and  reports.  Recent  efforts  have  included  greater  efforts  to  gather  data 
and  report  results.  Further  work  is  required.  Mission  objectives,  forces,  timelines,  MoMs, 
as  well  as  other  data  and  anal34ic  needs  should  be  articulated  so  that  future  reports  can 
provide  useful  information  for  the  future. 
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Operations  Other  Than  War: 

Summary  Observations 

•  Diverse  OOTW  add  to  complexity 

—  Of  the  mission  area 
—  Of  the  design  of  analysis  tools 
—  Of  C4ISR  tradeoff  development 

•  Mission  MoMs: 

—  Are  known,  but  vary  widely  by  mission 
—  Are  “standard”  for  many  trades  within  C4ISR 
—  Are  very  difficult  for  tradeoff  with  Forces 

•  Not  “ground  lost”  or  “casualty  ratio” 

•  Tradeoff  Analysis  focus  varies 

—  Operational-level  (JTF)  analysis  most  common 
—  PPBS/Acquisition-level  lacking 

•  Historical  databases  lacking 


Summary  Observations: 

The  working  group  made  several  general  observations  in  the  course  of  its  efforts.  Many  of  them 
support  the  preceding  discussion,  but  others  need  to  be  brought  out  separately. 

First,  we  reiterate  the  fact  that  the  diversity  of  OOTW  missions  adds  complexity  to  C4ISR  analysis. 
The  breadth  of  the  missions,  and  their  different  nature  adds  to  the  complexity  in  developing  general 
approaches  to  conducting  C4ISR  analysis  and  complicates  the  process  of  designing  tools  to  aid 
analysis.  Then,  we  observe  that  the  operational  experience  base  exists  to  allow  identification  or 
development  of  MoM.  These  MoMs  are  known,  but  vary  widely  by  mission  and  are  definitely  not 
“ground  lost”  or  “casualty  ratio.”  Many  C4ISR-related  trades  lead  to  MoMs  that  are  “standard” 
for  such  analysis.  However,  we  see  that  MOFE  measures  are  more  difficult  to  assess  in  tradeoffs 
with  other  systems  and  forces. 

We  also  note  that  we  found  it  natural  to  discuss  mission  and  tradeoff  analysis  in  the  context  of 
JTF-level  operations.  Most  of  our  discussions  focused  at  that  level.  We  see  that  these  trades  could 
easily  be  extended  to  acquisition-related  system  trades.  However,  for  PPBS  or  QDR-level  analyses, 
we  think  additional  methodology  is  necessary  since  no  single  OOTW  mission  is  likely  to  be  the  basis 
for  force  structuring  or  capability  analyses. 

Finally,  our  collective  experience  was  that  data  access  and  historical  data  to  support  operational 
planning  and  analysis  has  been  limited  and  limiting. 
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Operations  Other  Than  War: 

Recommendations 

•  Establish  an  effort  to  advance  our  knowledge  of  OOTW  and  C4ISR  in  OOTW 

—  Leading  to  analytic  frameworks 
—  Best  Practices 

•  Continue  efforts  to  ID  analytic  tool  requirements  to  assess  C4ISR  support  to 
OOTW 

•  Continue  development  of  methods  to  better  integrate  OOTW  analysis  into 
DoD  planning  &  programming 

•  Establish  a  common  syntax  &  semantics  to  facilitate  a  comparison  between 
study  results 


Recommendations: 

The  working  group  was  able  to  address  the  issues  specified  by  the  Terms  of  Reference  for  this 
meeting.  The  outline  for  generating  MoMs  and  structuring  C4ISR-centered  analyses  provides  a 
useful  step,  but  needs  to  be  extended  in  both  breadth  and  depth.  As  a  result,  we  make  the 
following  recommendations: 

Continue  this  process  of  methodology  development.  It  should  be  a  general  effort  to  advance  our 
knowledge  of  OOTW  and  C4ISR  in  OOTW,  leading  to  analytic  frameworks 
and  best  practices.  This  effort  should  also  continue  the  process  of  identifying  analytic  tool  to 
assess  C4ISR  support  to  OOTW. 

At  the  DoD  level,  this  effort  should  continue  development  of  methods  to  better  integrate  OOTW 
analysis  into  DoD  planning  &  programming.  This  need  largely  is  derived  from  the  fact  that  any 
single  OOTW  is  too  small  and  narrowly  focused  to  drive  force  structure  or  major  programming 
decisions. 

Another  aspect  of  these  developmental  efforts  is  to  establish  a  common  syntax  &  semantics  to 
facilitate  a  comparison  between  study  results.  Such  a  framework  will  facilitate  reviewing, 
comparing  various  missions  and  analyses,  and  will  encourage  more  broadly  based  analysis. 

Finally,  a  data  support  process  is  required  to  enable  routine  analysis.  Procedures  should  be  put  in 
place  to  ensure  data  is  gathered,  stored  and  made  available  for  analysis  efforts. 
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Operations  Other  Than  War: 

Recommendations  ( cant  ^d) 

•  Develop  a  deliberate  approach  to  data  collection 

—  ID  &  prioritize  needs 
—  Establish  collection  mechanism 
—  Establish  repositories 
—  Include  Non>DoD  sources 
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Analyzing  C4ISR  for  2010 
Infrastructure  Assurance 
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Infrastructure  Assurance: 

The  Problem 

•  How  does  C4ISR  support  Infrastructure  Assurance? 

-  WhatisC4ISR? 

—  What  is  infrastructure  assurance? 

•—  What  are  the  infrastructures? 

•  What  are  the  infrastructure  assurance  information  requirements? 

•  How  can  we  analyze  the  C4ISR  infrastructure  to  improve  support  of 
infrastructure  assurance  mission  area? 


Working  Group  Tasking: 

Here  is  the  problem  that  the  working  group  tried  to  address,  and  a  partial  answer: 

In  particular,  how  does  C4ISR  support  Infrastructure  Assurance? 

•  What  is  C4ISR  and  how  does  it  fit  into  the  context  of  Infrastructure 
Assurance? 

•  What  are  the  physical  and  information  requirements? 

•  And  how  can  we  analyze  and  use  the  C4ISR  infrastructure  to  improve  support 
of  the  Infrastructure  Assurance  mission  area? 

What  is  Infrastructure  Assurance? 

Infrastructure  Assurance  is  the  planning  to  improve  the  readiness,  reliability  and 
continuity  of  infrastructures  such  that  they  are: 

•  Less  vulnerable  to  disruptions  or  attack; 

•  Harmed  to  a  lesser  degree  in  event  of  disruption  or  attack;  and 

•  Can  be  readily  reconstituted  to  re-establish  vital  capabilities. 
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Infrastructure  Assurance: 

Characterize  C4JSR 

•  Infrastructure  Assurance/Critical  Infrastructure  Protection  (CIP)  is  new  and 
evolving  mission  area 

-  PDD  63  signed  May  1998 

—  Our  C4ISR  knowledge  from  conventional  warfare  does  not 
necessarily  apply  to  infrastructure  warfare 

•  Current  DoD  CIP  Infrastructure  Sectors 


•  Defense  Info.  Infrastructure 

•  Command,  Control  and 
Communications 

•  Intelligence,  Surveillance  and 
Reconnaissance 

•  Public  Works 

•  Financial  Services 


C4ISR  must  broaden  to  include  law  enforcement,  OGA  and  the  private 
sector 


Transportation 

Health  Affairs 

Emergency  Services 

Personnel 

Space 

Logistics 


Characterize  C4ISR: 


Infrastructure  Assurance,  until  recently  referred  to  as  Critical  Infrastructure  Protection 

(CIP),  is  a  relatively  new  and  evolving  mission  area: 

•  Presidential  Decision  Directive  (PDD)  63,  which  presents  the  overall  scope  of  Critical 
Infrastructure  Protection  and  calls  for  a  national  plan  and  structure  to  cope  with  the 
problem,  was  just  signed  in  May  of  this  year. 

•  It  addresses  a  new  kind  of  warfare,  unlike  what  we  have  experienced  in  conventional 
warfare. 

•  To  work  the  problem,  the  draft  DoD  CIP  Plan  divides  DoD  mission  areas  into  11 
infrastructure  sectors 

•  3  of  those  sectors  comprise  the  C4ISR  mission  area 

•  Each  of  the  3  C4ISR  sectors  must  address  all  the  physical,  as  well  as  cyber- 
related  assets  within  that  sector.  For  example  we  are  not  just  looking  at 
platforms  and  systems. 

•  Additionally,  these  sectors  cannot  be  considered  in  isolation  from  one  another. 
Each  is  often  dependent  on  the  other  to  achieve  its  mission  objectives,  (e.g., 
electric  power  to  run  the  computers,  water  to  cool  the  computers,  etc.)  These 
services  are  routinely  provided  by  commercial  assets  outside  the  government 
sector. 


Therefore,  the  C4ISR  community  must  broaden  its  vision  to  their  dependencies  on  other 
government  agencies  and  the  private  sector. 
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Assets,  Infrastructures,  and  Interdependencies: 

As  we  tried  to  demonstrate  here,  there  are  a  myriad  of  interdependencies  between  the 
critical  assets  within  certain  Defense  Infrastructures,  other  Defense  Infrastructures,  and 
the  Private/Commercial  Sector  Infrastructures. 

These  dependencies  and  value  chains  must  be  well  understood  for  analysis  within  the 
DoD  lifecycle  approach  to  the  CIP  problem  (as  defined  in  the  draft  DoD  CIP  Plan). 
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Infrastructure  Assurance: 

The  DoD  Strategy 

Approach: 

—  Identify  Relevant  Characteristics  of  Each  Infrastructure  that  is 
Critical  to  Military  Mission  Success 

—  Analyze  Military  Plans  to  Identify  Critical  Infrastructure  Assets 
—  Assess  Vulnerabilities  of  Critical  DoD  Assets 
—  Redirect  Resources  to  Reduce  Prioritized  Vulnerabilities 

—  Collaborate  with  Other  Government  Agencies  and  the  Private  Sector 
to  Reduce  Vulnerability  of  Private  Sector  Infrastructure  Assets 
Critical  to  DoD  Mission  Success 


The  DoD  Strategy: 

The  DoD  approach  to  the  CIP  or  Infrastructure  Assurance  problem  is  to: 

•  Identify  relevant  characteristics  of  each  infrastructure  that  is  critical  to  military 
mission  success. 

•  Analyze  military  plans  to  identify  critical  infrastructure  assets. 

•  Assess  vulnerabilities  of  critical  DoD  assets. 

•  Redirect  resources  to  reduce  prioritized  vulnerabilities. 

•  Collaborate  with  other  government  agencies  and  the  private  sector  to  reduce 
vulnerabilities  when  services  are  provided  from  outside  the  DoD  fence. 

This  should  be  done  in  the  context  of  six  (6)  lifecycle  phases  that  occur  before,  during 
and  after  events  which  may  compromise  or  disrupt  the  services  or  mission 
accomplishment  of  a  critical  infrastructure. 

Analysis,  Assessment  and  Remediation  are  proactive  preventive  measures  that  take  place 
continually.  Indications  and  Warning  primarily  occur  before  an  event.  Mitigation  occurs 
both  before  and  during  an  event.  Response  occurs  during  an  event  and  Reconstitution 
may  start  during  an  event,  but  is  generally  concentrated  afterward. 
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Infrastructure  Assurance: 

C4ISR  Must  be  Able  to  Support  the  lA  Life  Cycle 

Example;  Analvsis/Assessment 
Current  Requirements 

—  Architecture 

•  Example:  JCAPS,  ISAAC 

—  Data 

•  Commercial  Infrastructure  (Electric  Power,  Communications, 
Transportation) 

—  Example:  NCS  for  Communications 

•  Cyber  Infrastructure:  NIPC  Certs 
Future  Requirements 

—  Data 

•  Industry-Government  Information  Exchange 

—  Wargaming/Simulation 


C4ISR  Must  be  Able  to  Support  the  lA  Life  Cycle: 

This  and  the  next  graphic  are  examples  of  how  C4ISR  possibly  can  support  current  and 
future  infrastructure  assurance  requirements  within  a  given  lifecycle  phase. 

For  the  Analysis  and  Assessment  Phase,  operational  architecture  data  on  organizational 
missions,  functions,  structures,  systems  and  their  relationships  to  include  information 
flows  and  dependencies  can  possibly  be  provided  by  such  tools  as  the  Joint  C4ISR 
Architecture  Planing/Analysis  System  (JCAPS)  and  the  NCS  for  communications.  For 
Cyber  Infrastructure,  the  NIPC  and  various  Certs  are  C4ISR  components  that  support 
analysis  and  assessments.  Future  requirements  will  necessitate  an  exchange  of  industry 
and  government  information  for  wargaming  and  simulation  exercises. 
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Infrastructure  Assurance: 

C4ISR  Must  be  Able  to  Support  the  lA  Life  Cycle 

Example:  Indications  &  Warning 

Current  Requirements 

—  Inter-Agency  Information  Sharing 

•  Example:  MDITS  (Counter-terrorism) 

—  Indicators  (Criteria)  for  Minimum  Operating  Capability  for  Cyber 
Attack  only 

Future  Requirements 

—  Government — Commercial  Monitoring  &  Warning 

—  Indicators  (Criteria)  to  Assess  Status  of  Affected  Infrastructures, 
Operations,  or  Assets 

—  Trace  Ability 


C4ISR  Must  be  Able  to  Support  the  lA  Life  Cycle  (cont'd): 

For  the  Indications  and  Warning  Phase,  inter-agency  information  sharing  processes  will 
need  to  be  developed  to  satisfy  current  requirements.  MDITS  is  an  example. 

Future  requirements  will  necessitate: 

•  Government  -  Commercial  Monitoring  and  Warning 

•  Indicators  to  assure  status  of  affected  infrastractures,  operations,  or  assets,  and 

•  Trace  ability  to  the  root  of  the  problem. 
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Infrastructure  Assurance: 

Tools:  Current  Capability 

•  DoD  C4ISR  tools  generally  lack  the  capability  to  facilitate  detailed  lA 
analysis  because  C4ISR  requirements  generally  have  not  addressed  the  need 
to  support  Infrastructure  Assurance. 

•  LA  tools  have  been  developed  in  response  to  Infrastructure  Assurance 
community  requirements  and  are  effective  for  analyzing  commercial 
infrastructure  (power,  transportation,  communications,  and  information 
systems)  and  the  military  missions  they  support. 

•  Real-time  infrastructure  analysis  hindered  by  poor  integration  of  C4ISR 
and  lA  analysis  tools 


Tools;  Current  Capability 

Current  tools  capabilities  and  additional  tool  requirements  are  presented  on  this  and  the 
next  graphic. 

DoD  C4ISR  tools  are  currently  lacking  the  capability  to  facilitate  detailed  infrastructure 
analysis.  Therefore,  separate  tools  have  been  developed  to  meet  individual  or  specific 
needs.  These  tools  are  effective  in  the  Infrastructure  Assurance  community  for  analyzing 
power,  transportation,  and  communications.  However,  real-time  infrastructure  analysis  is 
hindered  by  little  or  no  integration  of  C4ISR  and  Infrastructure  Analysis  tools. 
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Infrastructure  Assurance 

Tool  Requirements 


•  lA  Tool  Needs: 

—  Integrated  and  Interoperable  Tools 

—  Comprehensive  Models  and  Tools  for  Analyzing  Defense 
Infrastructures 

—  Models  and  Tools  for  Interdependency  Analysis 

•  Need  Collaborative  Link  between  Infrastructure  Analysis  Centers  and 
Warfare  Centers 


Tool  Requirements: 

What  is  needed  are  integrated  and  interoperable  tools,  as  well  as  comprehensive  models 
for  analyzing  Defense  Infrastructures.  Models  and  tools  are  also  needed  to  conduct 
interdependency  analysis. 

We  also  need  to  establish  collaborative  links  between  infrastructure  analysis  centers  and 
warfare  centers. 
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Infrastructure  Assurance: 

Relative  Worth 

Possible  C4ISR  Contributions: 

•  Analysis  and  Assessment  Tools 

•  Near  Real  Time  Reporting  Across  All  Sectors 
-  I&W 

—  Infrastructure  Status 

•  Alternative  Courses  of  Action 

•  Synchronized  Information  that  Identifies  the  Criticality  Attribute  (e.g., 
Time)  of  Assets 

•  Information  Dissemination  and  Sharing  in  Support  of  Infrastructure 
Assurance 


Relative  Worth: 

As  suggested  above,  we  believe  the  C4ISR  community  can  provide  some  of  the  tools  for 
architecture  development  and  information  sharing  in  support  of  Infrastructure  Assurance. 
It  also  may  provide  some  of  the  tools  needed  for  near-real-time  reporting  across  all 
sectors,  particularly  with  respect  to  I&W  and  infrastructure  status.  Other  C4ISR  tools  are 
available  to  provide  information  on  alternative  courses  of  action  and  synchronized 
information  that  identifies  the  critical  attributes  (e.g.,  time)  of  assets. 
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Infrastructure  Assurance: 

Measures  of  Merit 


/National  \ 

/  Security  ^ 

Measuresof  Social  Continuity/Health  (e.g., 
Qualitative:  Population  ConHdence; 

Quantitative:  Delta  in  Dow  Industrial  Avg 

/Mission*  \ 

/  •  DoD  \  ; 

/  -  Other  Sectors  \ 

Measure  of  Mission  Effectiveness  (e.g.,  Success 
DoD’s  Ability  to  Mobilize) 

/  Infrastructtti^  ^ 

/  Functions 

Measures  of  Functional  Performance 
^  (e.g.,  Attack  Assessment;  Accuracy » 

\  :  Completeness) 

/  C4I$K  Systems 

■  Measure  of  System  Performance 

\  Timeliness,  Accuracy, 

Measures  of  Merit: 

We  suggest  four  hierarchical  measures  of  merit  with  respect  to  Infrastructure  Assurance 
and  its  dependence  on  C4ISR  as  a  basis  for  mission  success  and  national  security.  At 
each  level,  the  measures  of  merit  must  be  consistent.  Thus,  for  example.  Infrastructure 
Assurance  functions  (“the  lifecycle”  activities  of  Assessment,  Remediation,  I&W, 
Mitigation,  Response  and  Reconstitution)  depend  on  our  ability  to  perform  timely, 
complete,  acciuate  detection  and  assessment  of  an  attack.  To  do  this,  the  C4ISR  systems 
upon  which  Infrastmcture  Assurance  depends  must  meet  system  performance  measures, 
such  as 

•  timely; 

•  accurate; 

•  global  collection  of  threat;  and 

•  event  data; 

to  support  attack  assessment  and  reporting  back  to  affected  locations,  without  raising 
suspicions  of  the  attackers. 
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Infrastructure  Assurance: 

Key  Issues  and  Questions 

•  Definitions  &  Distinctions 

—  What  is  a  Critical  Asset? 

—  Infrastructure  Protection  vis-a-vis  Force  Protection 
—  Reliance  On  Host  Nation  Infrastructures 

•  Buy-in 

•  Data  -  “Achilles  Heel” 

•  Security 

•  Legal  and  International 


Key  Issues  and  Questions: 

Issues  within  lA  revolve  around  internal  discussions  and  decisions  such  as  definitions  and 
distinctions.  For  example,  what  really  is  lA?  Where  do  we  draw  the  line  between 
infi-astracture  protection  and  force  protection? 

Issues  that  are  internal  and  external  revolve  around  "Buy-In,"  that  is,  getting  our 
governmental  entities  both  within  DoD  and  other  Federal,  state  and  local  agencies 

•  to  share  information 

•  to  have  a  common  understanding  of  lA 

Buy-In  also  includes  government  to  commercial  industry  issues: 

•  sharing  proprietary  information 

•  who  pays? 

Issues  of  immediate  relevance  to  MORS  and  the  C4ISR  question  revolve  around  data, 
such  as: 

•  Fidelity 

•  Getting  Commercial  Sector  to  Provide  Data  (Often  Proprietary) 

•  Getting  Government  Organizations  to  Provide  Data  (Asking  vs. 
Tasking) 

•  Central  Data  Repository 
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And  of  course,  there  are  Security  issues  too  broad  and  too  deep  to  discuss  at  this  point, 
but  obvious  from  the  C4ISR  perspective  and  now  the  lA  perspective. 

There  is  a  substantial  CONUS-focused  legal  question.  And  there  are  substantial 
international  issues,  again  too  broad  and  too  deep  to  discuss  at  this  point,  but  relevant  to 
the  future  of  lA. 
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Infrastructure  Assurance: 

Recommendations 


•  MORS  continue  to  emphasize 
Infrastructure  Assurance  as  an 
essential  element  of  C4ISR  (and 
other)  analyses  and  wargaming 
and  simulation  exercises 


•  Joint  C4ISR  and  Infrastructure 
Assurance  community  analysis  of 
the  application  and  integration  of 
complementary  tools 


Recommendations: 


While  there  are  quite  a  few  recommendations  one  can  see  deriving  from  the  C4ISR-IA 
connection,  we  have  focused  on  two  (2). 

•  The  first  is  that  MORS  and  its  connections  to  the  education  community,  such  as 
the  National  Defense  University,  continue  to  actively  pursue  the  inclusion  of 
Infrastructure  Assurance  in  its  modeling/simulation  assessment  activities. 

•  And  the  next  is  that  we  in  the  lA  community  and  the  C4ISR  community  begin 
to  develop  mutual  understanding  of  our  requirements  in  order  to  develop 
integrated  tools  systems,  etc. 
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Infrastructure  Assurance: 

COBP  Analysis  Template 
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Analyzing  C4ISR  for  2010 
Peacetime  Engagement 
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Peacetime  Engagement  (PE): 

Working  Group  Members 


•  Bruce  Powers  (Chair) 

•  Dean  Free  (Co-Chair) 

•  Kevin  Brandt 

•  Trena  Covington 

•  LCDR  Marty  Keutel, 

•  Dr  Steve  Pilnick 

•  Dr  Dave  Schrady 

•  CDR  Mark  Tempestilli,  USN 

•  Chris  Vogt 


Office  of  CNO,N816 
Office  of  CNO,N816 
MITRE  Corporation 

Johns  Hopkins  University, 
Applied  Physics  Lab 

USN  Office  of  CNO,N6C 
EDO-Technology,  Inc. 
Naval  Postgraduate  School 
Office  of  CNO,N6C 

Navy  Warfare  Development 
Command 
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Peacetime  Engagement  (PE): 

Presentations 

**Maritime  Battle  Experiments^^  Chris  Vogt,  NWDC 

**Network  Centric  Warfare  Concepts^*’  Dean  Free,  N816 

‘Wavj  RMA  2020  Wargame  Results^  CDR  Bill  Valentine,  N81 

‘^Evaluating  Mission  Effectiveness  in  OOTW/*  Adam  Siegel,  CNA 

“Economic  Benefit  of  Forward  Presence’*  Dr  David  Schrady,  NFS 

“Design  Considerations/MOE  for  Linking  C4ISR  with  M&S,” 
Kevin  Brandt,  MITRE 

“Net  Collective  C^,”  CDR  Randy  Bowdish,  Office  of  CNO,  N51 
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Peacetime  Engagement  (PE): 

Definition 

The  application  of  resources  (including  forces)  in  peacetime  to  make  shooting  less 
likely.  Successful  PE  would  promote  other  goals: 

•  Stability  of  commerce 

•  Deterrence 

•  Separation  of  potential  belligerents 

•  Exercises  with  (potential)  allies 


Definition: 

•  No  current  Joint  Staff  definition  of  Peacetime  Engagement  (PE). 

•  Current  related  Joint  Staff  definitions: 

-  Overseas  Presence:  The  totality  of  US  military  instruments  of  power  deployed 
overseas.  It  is  that  portion  of  the  force  postured  overseas  to  support  performing 
the  full  range  of  military  activities  comprised  of  permanent  forces,  temporary 
forces  and  infrastructure. 

-  Enpgement:  All  military  activities  involving  other  nations  intended  to  shape  the 
regional  security  environment  in  peacetime.  Engagement  is  comprised  of 
operations,  exercises  and  other  foreign  military  interactions. 

•  The  above  definition  characterizes  PE  as  deterrent  in  nature  and  includes  benefits  of 
deterrence  as  broad  goals  that  are  measurable  to  some  degree. 
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Peacetime  Engagement  (PE): 

Pre-Conflict  C4ISR  Performance  Measures 

•  Threat  located 

-  %  of  total  OOB 

-  %  of  number  deployed  OOA 

-  %  in  vicinity  of  blue  opareas 

-  %  threat  weapons  close 

-  False  target  rate  impact 

•  %  needed  info  available 

—  At  partners’  command  centers 
—  At  US  command  centers 


Pre-Conflict  C41SR  Performance  Measures: 

These  represent  a  sample  of  C4ISR  measures  in  the  pre-conflict  phase  of  a  campaign. 
One  would  dso  need  to  consider: 

•  Threat  intentions 

•  Threat  strategy 

•  Centers  of  gravity  for  the  threat 

•  Courses  of  action  to  deter  the  threat  —  need  to  understand  our  options  and  the 
threat’s  response;  “effects-based”  options,  not  just  an  attrition-based  approach 

Bottom  line:  ISR  is  MUCH  more  than  a  collection  of  numbers. 
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Peacetime  Engagement  (PE): 

Lessons  Learned 


Discussion  led  to  the  list  of  lessons  learned  on  this  and  the  next  two  slides.  The 
discussion  centered  around  the  insights  provided  by  the  seven  briefings  received  by  the 
working  group,  plus  discussions  of  the  purpose  and  objectives  of  the  workshop. 
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Peacetime  Engagement  (PE): 

Lessons  Learned  (confd) 


Intel  prep  of  battleaeld  is  conducted  during  PE 
PE  provides  means  to  partially  validate  models  of  warfare 
PE  not  a  lesser  included  case  of  warfare 

Greater  use  and  size  of  C4I  systems  may  bring  greater  C4I  vulnerability 
Deploying  C4ISR  increasingly  follows  model  of  naval  deployment 


E-7 


Peacetime  Engagement  (PE): 

Coalitions 

•  Greater  number  of  potential  partners  in  PE  than  in  specific  MTWs 

•  Interoperable  C4I  required  for  coalitions 

•  Reliance  on  coalition  wartime  capabilities  requires  peacetime  investment  in 
C4I 

•  Coalition  interoperability  improves  war  fighting  capability 


Coalitions: 

These  are  conclusions  about  coalitions  that  resulted  from  discussion  of  PE  in  the  broader 
context. 
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Peacetime  Engagement  (PE): 

Key  Issues  and  Questions 

1.  After  Cold  War,  the  spectrum  of  potential  conflict  has  broadened 

-  The  probability  of  occurrence  of  events  on  this  spectrum  may  be 
differently  distributed 

-  Data  collection  and  analysis  of  that  issue  should  follow 

2.  Naval  Postgraduate  School  research  suggests  major  benefits  ($10Bs)  in 
savings  on  oil  prices  by  prompt  deployment  of  naval  forces  in  crises.  Can 
the  scale  of  benefits  for  deployment  of  C4ISR  be  similarly  identified  (e.g., 
AWACS  deployment  impacting  prices  of  drugs  from  Colombia)? 


Key  Issues  and  Questions: 

1.  The  nature  of  the  world,  the  threat,  has  changed  drastically  since  the  end  of  the  cold 
war.  The  distribution  of  risk  of  warfare  is  quite  different,  with  more  uncertainty 
about  the  probabilities  of  specific  threats.  There  is  a  need  to  collect  data  and  develop 
the  distribution  of  events  on  the  current  and  future  spectra  of  conflict. 

2.  The  Naval  Postgraduate  School  study  identified  significant  potential  savings  on  oil 
prices  by  early  deployment  of  naval  forces.  It  is  possible  that  early  deployment  of 
C4ISR  systems  could  provide  similar  benefits. 
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Peacetime  Engagement  (PE): 

Key  Issues  and  Questions  (confd) 

3.  The  less  an  event  looks  like  war,  the  less  able  we  are  to  articulate  its 
governing  processes.  C4ISR  in  PE  requires  drawing  in  political 
constraints/considerations  to  a  degree  greater  than  for  other  points  on  the 
spectrum.  This  suggests  peculiar  data  requirements  for  military/C4ISR 
commanders  in  PE  settings. 

4.  For  PE  events,  the  cyclical  process  (deHne  MOM,  measure  sensitivity, 
collect  data,  evaluate  data,  report)  needs  to  also  capture  critical  C4ISR  data 
and  effects. 
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Peacetime  Engagement  (PE): 

Recommendations 

•  Develop  broadly-accepted  TOR  for  PE:  definitions,  objectives,  goals, 
distinguishing  relationships  from 

OOTWtoMTW 

•  Develop  CSOP  for  PE 

*  Develop  MoMs  and  data  collection  checklists  for  PE 

*  Expand  JULLS  formats  and  databases  to  include  C4ISR  data 


Recommendations: 

•  Recognizing  PE  as  an  important  mission  area,  with  attendant  principles  and 
distinguishing  relationships,  will  assist  in  identifying  the  peculiar  tasks  of  PE,  and 
gamering  funding  for  their  successful  undertaking. 

•  Common  Standard  Operating  Procedures  must  account  for  the  many  differences 
(historical,  cultural,  financial,  educational,  etc.)  that  exist  in  the  many  countries  that 
we  engage  during  PE  operations.  “One  size  fits  all”  is  not  the  answer.  We  need  to 
KNOW  our  PE  “targets”  much  better  and  tailor  each  engagement  to  that  particular 
country.  Specific  C4ISR  exchanges  are  good  candidate  elements  for  tailored  PE. 

•  Expansion  of  the  Joint  Uniform  Lessons  Learned  database  to  include  C4ISR  data  and 
information  will  help  long  term  analysis  of  C4ISR  issues. 
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Peacetime  Engagement  (PE): 

Recommendations  (confd) 

•  Study  the  impact  of  deploying  C4ISR  in  peacetime 

—  Assess  utility  of  C4ISR  deployment  on  its  own 

—  Assess  how  early  C4ISR  deployment  facilitates  early  combat  power 
by  follow-on  forces 


Recommendations  (cont’d): 

The  results  of  such  a  study,  successfully  completed,  should  help  determine  the  value 
added  by  C4ISR  systems.  The  study  should  also  address  how  a  robust  C4ISR  exchange 
program  helps  build  coalitions  and  therefore  serves  as  a  significant  deterrent. 
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Peacetime  Engagement  (PE): 

Assessment  of  Current  State-o  fthe-Practice 


Assessment  of  Current  State-of-the-Practice: 

Scenarios  represent  the  only  good  news  on  this  slide.  In  all  other  areas,  assessment  of  the 
current  state-of-the-practice  is  discouraging.  There  is  much  work  to  be  done  to  develop 
measures  of  effectiveness  and  tools  and  to  collect  data  relevant  to  proper  analysis  of 
C4ISR  issues  related  to  PE. 
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Peacetime  Engagement  (PE): 

Summary 

KEY  ISSUES 

•  Since  the  cold  war,  the  spectrum  of  potential  conflict  has  broadened.  The 
probability  of  occurrence  of  events  on  this  spectrum  may  be  differently 
distributed.  Data  collection  and  analysis  of  that  issue  should  follow. 

•  The  less  an  event  looks  like  war,  the  less  able  we  are  to  articulate  its  governing 
processes.  C4ISR  in  PE  requires  drawing  in  political  constraints/  considerations 
to  a  degree  greater  than  for  other  points  on  the  spectrum.  This  suggests  peculiar 
data  requirements  for  military/  C4ISR  commanders  in  PE  settings. 

•  For  PE  events,  the  cyclical  process  (define  MOM,  measure  sensitivity,  collect  data, 
evaluate  data,  report)  needs  to  also  capture  critical  C4ISR  data  and  effects. 

RECOMMENDATIONS 

•  Develop  broadly-accepted  TOR  for  PE:  definitions,  objectives,  goals, 
distinguishing  relationships  from  OOTW  to  MTW 

•  Develop  CSOP  for  PE 

•  Develop  MoMs  and  data  collection  checklists  for  PE 

•  Expand  JULLS  formats  and  databases  to  include  C4ISR  data 


Summary: 

•  KEY  ISSUES 

•  The  nature  of  the  world  and  the  threat,  has  changed  since  the  end  of  the  Cold 
War.  The  distribution  of  risk  of  warfare  is  quite  different,  with  more  uncertainty 
about  the  probabilities  of  specific  threats.  There  is  a  need  to  collect  data  and  develop 
the  distribution  of  events  on  the  current  and  future  spectra  of  conflict. 

•  RECOMMENDATIONS 

•  Recognizing  PE  as  an  important  mission  area,  with  attendant  principles  and 
distinguishing  relationships,  will  assist  in  identifying  the  peculiar  tasks  of  PE  and 
gamering  funding  for  their  successful  undertaking. 

•  Common  Standard  Operating  Procedures  must  account  for  the  many  differences 
(historical,  cultural,  financial,  educational,  etc.)  that  exist  in  the  many  countries  that 
we  engage  during  PE  operations.  “One  size  fits  all”  is  not  the  answer.  We  need  to 
KNOW  our  PE  “targets”  much  better  and  tailor  each  engagement  to  that  particular 
country.  Specific  C4ISR  exchanges  are  good  candidate  elements  for  tailored  PE. 

•  Expansion  of  the  Joint  Uniform  Lessons  Learned  (JULLS)  database  to  include  C4ISR 
data  and  information  will  help  long  term  analysis  of  C4ISR  issues. 
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Analyzing  C4ISR  for  2010 
Architectures 
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Overview: 


Analytical  paradigms  have  traditionally  focused  on  evaluating  the  effectiveness  of  a 
platform  or  system.  Architectures  provide  a  structured  approach  for  describing  and 
understanding  a  system-of-systems  associated  with  a  specified  domain,  such  as  C4ISR 
and  in  doing  so  allow  that  domain  to  he  considered  as  if  it  were  an  entity.  Using  the 
architecture  construct  enables  our  analytical  focus  to  evolve  from  being  platform¬ 
centric  to  become  architecture-centric. 
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Architectures:  Current  Status 


Architectures  are  an  evolving  discipline 

-  part  science  and  part  art 

PROS 

•  Provide  an  understanding  of  how  information  is  used  to  accomplish  work 
and  how  Information  Technology  supports  work 

•  Form  the  basic  data  and  data  relationships  that  can  support  C4ISR  issue 
analysis  through  various  analytical  techniques  to  include  Modeling  and 
Simulation 

CONs 

•  Generally  are  static  snapshots  in  time  and  hand  crafted;  difHcult  to  reuse 


"  i^ar^^techK-e  should  riot  be  ah;  dnd  m^  , 

?' ,^w.but:shoui<l.be.a'meaiis_|p rinond. ' 
ShouM  have  a  well-defined  purpose  arid  objective. 

,  stated  ^  the  beginning  of  the  effort  I 


Current  Status: 

While  major  architectural  precepts  are  well  defined  and  generally  well  accepted,  architectures 
are  a  still  evolving  discipline  which  demand  a  combination  of  both  art  and  science. 

From  the  positive  perspective,  architectures  help  us  understand  the  relationship  and 
dependencies  among  tasks,  operational  elements  accomplishing  those  tasks  and  the  information 
flow  that  needs  to  occur  among  the  operational  elements  to  accomplish  the  tasks  appropriately. 
They  enable  us  to  describe  the  systems  and  conununications  that  support  that  information  flow 
across  nodes  and  in  doing  so,  enable  the  understanding  of  how  Information  Technology 
supports  work. 

Architecture  descriptions  form  the  basic  data  set  of  activities,  operational  elements, 
information  flows,  nodes,  systems  and  communications  that  support  a  wide  variety  of  C4ISR 
issue  analyses  using  various  techniques  to  include  Modeling  and  Simulation. 

The  use  of  architectures  may  be  limited  or  restricted  because  they  generally  provide  a  static 
perspective  and  represent  snapshots  in  time.  They  traditionally  have  to  be  developed  on  an 
individual  basis  and  the  information  is  difficult  to  capture  for  reuse  in  related  efforts. 
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Architectures  have  too  frequently  been  seen  as  an  end  in  themselves  when  their  true 
value  is  as  a  means  to  an  end.  A  successful  architecture  effort  must  begin  with  a  well- 
defined  purpose  and  objective. 
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Architectures: 


*  Architectures  can  provide  a  seamless  bridge  between  the  C4ISR 
user  and  developer  communities. 

*  Operational  architecture  can  capture  operational  requirements 
in  a  fashion  that  allows  consistent  traceability  to  design 
specifications. 

*  Symbiotic  relationship  between  analysis  and  architectures 
which  need  to  be  emphasized  and  strengthened 


Potential: 

The  traceable  linkage  across  the  architecture  products  provide  an  audit  trail  relating 
cind  translating  mission  effectiveness  requirements  of  the  C4ISR  user  to  system 
requirements  to  which  the  system  developer  must  build. 

Because  of  that  traceable  linkage,  operational  architectures  are  able  to  capture 
operational  requirements  in  a  way  such  that  design  specifications  can  be  consistently 
traced  to  those  requirements. 

The  relationship  between  architectures  and  analyses  using  a  variety  of  techniques  is 
only  beginning  to  be  explored.  The  close  and  mutually  beneficial  relationship  between 
architecture  and  analysis  need  to  be  emphasized  and  strengthened. 
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Architectures:  Keys  to  Realizing  Potential 

•  Must  be  able  to  access,  reuse  and  manipulate  data  across  a  wide  range  of 
stakeholders 

•  Users 

•  Developers 

•  Acquirers 

•  Must  integrate  human  factors  into  architecture:  People  are  an  integral 
part  of  architectures 

•  Need  to  represent  threat  more  overtly  to  include  analysis  of  adversary  as 
well  as  friendly  (interactive) 

•  Tools  are  critical 

•  Better  visualization 

•  Greater  flexibility 

•  More  usable  by  the  operational  subject  matter  experts 

•  Must  be  able  to  better  articulate  the  “so  what”  (operational  impacts)  of 
architectural  analysis  to  warfighters  and  to  decision  makers 


Keys  to  Realizing  Potential: 

In  order  to  realize  the  full  potential  of  architectures,  the  various  stakeholders  from 
system  users  to  developers  and  acquirers,  must  be  able  to  access  the  architecture  data, 
manipulate  it  and  reuse  it  in  a  variety  of  ways. 

Current  architecture  constructs  do  not  overtly  recognize  and  portray  the  impact  of 
human  factors.  Because  people  are  an  integral  part  of  any  architecture,  architecture 
constructs  need  to  be  expanded  to  more  specifically  represent  human  factors. 

Threat  also  needs  to  be  represented  more  overtly  in  architectures. 


Getting  data  from  a  hardcopy  text  presentation  to  a  structured  softcopy  presentation  is 
essential  and  will  enable  the  use  of  automated  tool.  The  use  of  tools  is  critical  to 
provide  better  visualization,  greater  flexibility  and  to  make  the  data  more  usable  to 
operational  subject  manner  experts. 


Architecture  description  developers  must  perceive  that  their  primary  objective  is  not 
just  the  accumulation  and  presentation  of  the  architecture  data  but  the  analysis  of  that 
data  to  identify  the  "so  what",  i.e.  the  operational  impacts  and  to  better  articulate  those 
impacts  to  warfighters  and  decision  makers. 
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Architectures:  Measures  of  Merit 


•  Still  very  immature,  promising  efforts  at  early  stages 

•  Must  be  able  to  establish  quantiHable  relationship  between 
measures  of  performance  and  operational  measures  of 
effectiveness/outcome 


Key  to  ^chitecture-Centric  Analysis 
Still  tend  to  Aink  in  terms  of  platfoi^, 
must  leaiti  how  to  tr^fer  that  to  system-of-systenM; 


Measures  of  Merit: 

Currently  there  are  no  well-conceived  and  realized  measures  of  merit  associated  with 
architectures. 

There  are  three  aspects  of  architectures  which  potentially  could  be  measured: 

(1)  how  well  the  architecture  description  has  been  developed, 

(2)  to  what  extent  is  the  architecture  description  being  used  and 

(3)  the  value  of  the  architecture  in  achieving  C4ISR  goals  and  objectives. 

The  third  aspect  is  considered  the  most  meaningful  and  the  most  difficult  to  quantify. 
It  is  the  aspect  most  addressed  in  ongoing  efforts.  A  metric  for  this  aspect  must  be 
able  to  establish  a  quantifiable  relationship  between  measures  of  performance  for 
individual  systems  and  operational  measures  of  mission  effectiveness  associated  with 
the  architecture’s  system-of-systems. 

The  community  appears  to  just  be  in  the  initial  stage  of  addressing  metrics  for 
architectures.  While  current  efforts,  such  as  those  by  John  Hopkins,  Boeing  and 
OASD  (C3I),  are  very  immature  but  they  show  promise. 

The  ability  to  measure  the  value  of  a  specified  architecture  in  achieving  mission 
effectiveness  is  key  to  architecture-centric  analysis. 
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Architectures:  Assessment 


Human  Factors 


Availability 


Acqnisihon  Survivability  (lA) 
Analysis 


Description 


Potential 

Relevancif 


Assessment: 


The  above  graphic  provides  a  color-coded  assessment  of  the  current  state  of  the  art  for 
architectures.  There  is  a  well-defined  methodology  in  place  —  the  C4ISR  Architecture 
Framework  has  been  mandated  for  use  in  DoD. 

While  there  is  some  capability  to  portray  human  factors  via  architectures,  it  is  not  well  defined. 
In  terms  of  the  human  factor  aspect  of  architecture  development,  there  is  a  need  for  a  greater 
ability  for  collaborative  development. 

Scenarios  can  be  portrayed  in  architectures  but  additional  robustness  and  flexibility  is  needed. 

Measures  of  Merit  are  only  just  now  beginning  to  be  addressed. 

While  there  are  a  variety  of  tools  available,  a  coherent  tool  set  is  yet  to  be  achieved. 

Architecture  data  is  well  defined,  but  there  is  no  centrally-managed  or  maintained  architecture 
data  repository.  Thus  gathering  data  for  architecture  descriptions  generally  requires  a  great 
deal  of  primary  collection  effort. 

Architectures  have  the  capability  of  reducing  risk  in  acquisition  efforts  and  supporting  analysis 
of  information  assurance,  but  generally  are  not  used  to  the  extent  possible. 

Developers  are  generally  quite  good  at  presenting  the  architecture  data  in  a  report  but  are 
frequently  not  successful  in  communicating  the  "so  what"  of  an  architecture. 
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Architectures:  Summary 


•  Architectures  are  an  evolving  discipline 

-  part  science  and  part  art 

•  Methodology  well  defined  but  still  evolving 

•  Potential  for  architecture-centric  analysis 

-  structured  data  repository 

-  automated  tools 

•  Need  greater  emphasis  on  the  “so  what” 

•  Measures  of  Merit  still  in  very  early  stages  but  some 
promising  beginnings 


Summary: 

Architectures  are  an  evolving  discipline  which  has  reached  a  certain  level  of  maturity 
but  will  continue  to  evolve.  Developing  architecture  descriptions  requires  both  science 
and  art. 

While  the  methodology  is  well  defined  in  the  C4ISR  Architecture  Methodology,  it  will 
continue  to  evolve  based  on  use  across  DoD. 

There  is  great  potential  for  architecture-centric  analysis  but  a  structured  data  repository 
and  automated  tools  are  critical  to  achieve  that  potential. 

Architectures  must  not  be  static  descriptions  but  require  greater  emphasis  of  the  "so 
what." 

Measures  of  Merit  are  in  very  early  stages  but  there  are  promising  beginnings  across 
the  community. 
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Analyzing  C4ISR  for  2010 
Analytical  Techniques  and  Tools 


This  is  the  Final  Out  brief  from  the  Analytical  Techniques  and  Tools  Working  Group.  Dr 
A1  Brandstein  was  the  Chair  and  Dr  Roy  Rice  was  the  Co-chair.  Members  of  the 
Working  group  are  listed  in  the  Attendees  list.  Special  thanks  goes  to  Mr  Steve  Ritacco 
and  Captain  Geof  Maron  for  their  superb  work  in  recording  the  discussions  and  preparing 
this  briefing. 
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Preparatory  remarks  are  included  in  the  Introduction.  This  details  our  mission  and 
objectives  for  our  Working  Group.  It  also  outlines  the  context  by  which  we  focused  our 
discussions.  Next  is  a  detail  of  the  taxonomy  we  devised  to  frame  how  we  discussed  the 
uses,  misuses,  deficiencies  and  opportunities  for  improvement  for  our  analysis 
methodologies  and  tools.  Finally,  we  summarize  our  two  days  of  discussions  with  a  list 
of  findings. 
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Analytical  Techniques  and  Tools: 

Mission  /  Objectives 


Understand  elements  of  C41SR 

Discuss  wide  range  of  analytical  methodologies 

Focus  on  Decision  Issues 

Match  methodologies  to  issues 

Determine  deficiencies/opportunities 


Mission  /  Objectives: 


Our  first  objective  was  to  come  to  some  common  understanding  of  the  elements  of 
C4ISR.  This  was  from  an  operational,  analytical,  and  modeling  standpoint.  This  allowed 
us  to  focus  on  the  analysis  needs.  Then  we  received  several  briefings  covering  a  wide 
range  of  C4ISR  topics  and  analysis  approaches.  With  this  range  of  applicable 
methodologies  in  mind,  we  discussed  the  various  decision  issues  or  questions  that 
decision  makers  are  asking  now,  or  will  probably  be  interested  in,  in  the  future. 


Our  next  effort  was  to  match  some  of  the  relevant  methodologies  to  the  particular 
decision  issues.  This  then  allowed  us  to  discuss  some  of  the  specific  deficiencies  or 
opportunities  to  improve  the  state-of-the-art. 
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Analytical  Techniques  and  Tools: 

Actions  of  Group 

•  Methodologies 

—  JWARS;  NETWARS;  Optimization;  Complex  Adaptive 
Systems;  Systems  Dynamics;  etc. 

—  Discussions 

•  Develop  taxonomy  of  Decision  Issues 

•  Rate  the  “Starr”  Chart 

•  Glean  pearls  of  wisdom 


Actions  of  Group: 

The  first  day  of  the  workshop,  we  heard  presentations  from  several  attendees  on 
analytical  methodologies  they  are  using,  modeling  efforts  they  are  involved  in,  and 
current/future  capabilities.  These  were  briefings  by: 

-  LTC  Dan  Maxwell  -  JWARS 

-  LTC  Pat  Dye  -  NETWARS 

-  Roy  Rice  -  C4ISR  Optimization  using  the  Sensor  Platform  Allocation  Model 
(SPAM) 

-  Capt  Geoff  Maron  -  an  application  of  Complex  Adaptive  Systems  (CAS) 

-  Charlie  Hall  -  Linking  C4ISR  to  Military  Effectiveness 

-  Steve  Upton  -  Generative  Analysis:  Using  Composable  Agent-based  Simulations  for 
Future  C4ISR  Concept  Exploration 

-  Steve  Kolek  -  Battlefield  Communications  Network  Modeling 

-  Robie  Samanta  Roy  -  Analyzing  C4ISR  on  Strike  Mission  Performance 

-  Evan  Ellis  -  Systems  Dynamics  Modeling  to  Represent  C4ISR 

The  discussions  that  followed  these  presentations  led  us  to  develop  a  taxonomy  by  which 
to  address  many  of  the  Decision  Issues.  This  provided  a  context  by  which  we  rated  the 
various  areas  of  the  chart  Dr  Starr  provided  as  a  guideline  in  his  plenary  presentation. 
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Context  of  C4ISR  Modeling: 


The  discussions  quickly  led  us  to  conclude  that  our  analyses  of  relevant  C4ISR  “systems” 
includes  C4ISR  “systems”  as  independent  variables  and  dependent  variables.  What  we 
mean  here  is  that  we  often  perform  analyses  of  the  specific  elements  of  C4ISR.  We 
compare  alternative  sensors,  platforms,  architectures,  and  their  performance.  This  is 
where  C4ISR  “systems”  could  be  considered  as  independent  variables.  There  are  also  the 
cases  where  we  analyze  other  parts  of  our  application  of  military  force  (forces,  tactics, 
doctrine,  etc.)  where  C4ISR  “systems”  must  play  but  are  not  the  analytical  topic  of 
interest.  They  must  be  represented  and  accounted  for  properly  in  our  analyses  and 
models.  This  is  where  they  would  be  considered  dependent  variables. 


In  that  context,  we  used  the  depiction  here  to  represent  the  elements  of  the  C4ISR 
“system”  from  an  operational,  analytical,  and  modeling  perspective. 
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Category  of 

Decision 

Issues 


Analytical  Techniques  and  Tools: 

Areas  of  C4ISR  Analysis 


Decision  Makers 
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Analytical  Techniques  and  Tools: 

mtUn  Each  Area  (Cube) 
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Analytical  Techniques  and  Tools 

Acquisition  Slice 


C 

^  Force  Structure 

e 

g  Cognitive  Behavior 

o 

r  CONOPS 

i 

®  Architecture 

s 

Vulnerabilities 


Objectives 

MTW  SSC  OOTW 


O  QTQ 


Analytical  Techniques  and  Tools 

Operational  Slice 


C 

®  Force  Structure 

t 

e 

Cognitive  Behavior 

CONOPS 

Architecture 
s 

Vulnerabilities 
Trade  Space 


Objectives 

MTW  SSC  OOTW 
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Analytical  Techniques  and  Tools: 

POM  /  Budget  Slice 


c 

^  Force  Structure 

e 

g  Cognitive  Behavior 

o 

r  CONOPS 

i 

®  Architecture 

s 

Vulnerabilities 
Trade  Space 


Objectives 

MTW  SSC  OOTW 
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Analytical  Techniques  and  Tools 

Acquisition  /  Force  Structure 


MTW  SSC  OOTW 

Force  Structure 

.  MTW  &  SSC 

-  Room  for  im provem ent 

-  Scenarios  rated  Red,  D  ata  rated  Red 
.  OOTW 

-  Data  lacking 

-  Gaping  void  below  high  level  documents 

-  Tools  cannot  be  found  even  if  objectives  are  ID’d 

-  No  architecture  exists  for  interagency  C  ^(NGO  &  GO) 


Analytical  Techniques  and  Tools 

Acquisition  /  Cognitive  Behavior 


Analytical  Techniques  and  Tools: 

Acquisition  /  CONOPS 

MTW  SSC  OOTW 


CONOPS 


.  MTW  /  SSC 

-  Doctrine  is  wired  into  the  tools 

-  C4  deficiencies  exist  within  the  tools 
•  OOTW 

-  T  ools  are  lack  in  g 
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Analytical  Techniques  and  Tools: 

Acquisition  /  Architecture 


MTW  SSC  OOTW 

Architecture 

•  Data  is  difficult  to  obtain 

-  Especially  in  OOTW  (NGO  and  coalition) 
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Analytical  Techniques  and  Tools: 

Acquisition  /  Vulnerabilities 


Vulnerabilities 


MTW  SSC  OOTW 


•  Computer  security  /  information  warfare  tools  are  required 

-  Not  much  interest  in  this  area 

-  We  have  little  experience  or  basis  for  creating  these  tools 


Analytical  Techniques  and  Tools: 

Acquisition  /  Trade  Space 


MTW  SSC  OOTW 


Trade  Space 


•  No  tools  exist  for  architecture  trade  space 

•  OOTW 

-  Data  lacking 

-  Gaping  void  below  hi^  level  documents 

-  No  architecture  exists  for  interagency  C  2  (NGO&  GO) 
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Analytical  Techniques  and  Tools: 

Operational  /  Force  Structure 


Force  Structure 


MTW  SSC  OOTW 


•  Greenish  tint  to  the  yellow 

-  This  is  the  area  in  which  we  have  the  most  expertise 

•  Don’t  model  ambiguity  well 

~  We  know  we  don’t  do  this 

•  Data  remains  a  problem 

“  Availability 

-  Abstraction 
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Analytical  Techniques  and  Tools 

Operational  /  Cognitive  Behavior 


MTW  SSC  OOTW 

llllllllll^^ 


•  This  area  has  significant  potential 

-  Effects  of  stress 

•  Continue  development  of  tools  in  this  area 

-  Measures  to  exploit  our  experiences 

•  Need  data  collection  methodology 


G-18 


Analytical  Techniques  and  Tools: 

Operational  f  CONOPS 

MTW  SSC  OOTW 


CONOPS 


•  We  know  methodologies 

•  Don’t  have  the  resolution  in  our  tools  to  provide  sufficient  insights 
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Analytical  Techniques  and  Tools: 

Operational  /Architecture 


MTW  SSC  OOTW 


Architecture 


•  Requires  concentration  of  near  term  investment 

•  Measures  of  Merit  required  to  feed  to  operational  decision  makers 

•  Tools  need  to  be  dynamic 

-  Economic  pressures 
—  COTS  impact  on  force  structure 
•  Commercial  satellite  systems, . . . 
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Analytical  Techniques  and  Tools: 

Operational  /  Vulnerabilities 


MTW  SSC  OOTW 


Vulnerabilities 

•  Need  to  be  able  to  study  the  sensitivities  of  our  architectures  to  our 
assumptions 
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Analytical  Techniques  and  Tools: 

POM/Biidget  /  Force  Structure 


MTW  SSC  OOTW 


Force  Structure 


•  Low  level  MoM  acceptable,  but  aggregate  measures  are  poor 

•  Scenarios  fair  for  high  levels,  poor  for  low  levels 

•  Tools  are  good  at  mission  level 

•  Tools  are  poor  at  the  joint  level,  across  missions,  and  among  services 

•  Data  is  fair 
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Analytical  Techniques  and  Tools: 

POM/Budget  /  Cognitive  Behavior 


MTW  SSC  OOTW 

Cognitive  Behavior 

•  Ability  in  area  is  in  infancy  stages 

•  Data  is  poor 

•  Tools  are  poor 

•  Measures  are  fair 


Analytical  Techniques  and  Tools: 

POM/Budget  /  CONOPS 


MTW  SSC  OOTW 

CONOPS  lllllllllllllll^^l^^ 

•  CONOPS  specified  enough  for  budgeting  purposes 


—  Consideration  of  broader  scenarios  is  poor 
•  Tools  may  be  poor,  but  data  and  measures  are  good 
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Analytical  Techniques  and  Tools: 

POM/Biidget  /  Architecture 


Architecture 


MTW  SSC  OOTW 


•  No  agreement  on  MoMs  across  scenarios,  services,  and  different  C4ISR 
systems 

•  Tools  are  good 

•  Data  is  good  to  fair 
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Analytical  Techniques  and  Tools: 

POM/Budget  /  Vulnerabilities 


MTW  SSC  OOTW 


Vulnerabilities 


•  Questions  about  levels  of  redundancy  and  levels  of  fitness  are  not  well 
described 

•  Tools  that  show  the  effects  of  non-linear  interactions  and  the  ability  to 
measure  their  impact  on  complex  systems  are  lacking. 

•  Data  is  poor 

•  MOMs  are  poor 

•  Tools  are  fair 
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Analytical  Techniques  and  Tools 

POM/Budget  /  Trade  Space 


MTW  SSC  OOTW 

Trade  Space 

•  Good  to  fair  within  mission  and  functional  area 

•  Poor  to  fair  between  missions  and  services 

•  MoMs  fair 

•  Tools  fair 

•  Data  poor 
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Analytical  Techniques  and  Tools: 

Matching  Issues  and  Methodologies 


Technology  ->  New  Doctrine/ 
tactics 


Vulnerabilities 
Trade  Space 
Cognitive  Behavior 


Campaign  Level  Sim 
Mission  Level  Sim 
Engagement  Level  Sim 

Optimization 
ame  theory 
AS 

Dynamic  Programming 
Functional  Decomposition 
Systems  Dynamics 
W  argaming 
...  etc. 
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Analytical  Techniques  and  Tools: 

Coalitions 


Greater  number  of  potential  partners  in  PE  than  in  specific  MTWs 
Interoperable  C4I  required  for  coalitions 

Reliance  on  coalition  wartime  capabilities  requires  peacetime  investment 
inC4I 


Coalition  interoperability  improves  war  fighting  capability 


Coalitions: 


These  are  conclusions  about  coalitions  that  resulted  from  discussion  of  PE  in  the  broader 
context. 
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Analytical  Techniques  and  Tools: 

Key  Issues  and  Questions 

1.  After  Cold  War,  the  spectrum  of  potential  conflict  has  broadened 

-  The  probability  of  occurrence  of  events  on  this  spectrum  may  be 
differently  distributed 

-  Data  collection  and  analysis  of  that  issue  should  follow 

2.  Naval  Postgraduate  School  research  suggests  major  benefits  ($10Bs)  in 
savings  on  oil  prices  by  prompt  deployment  of  naval  forces  in  crises.  Can  the 
scale  of  benefits  for  deployment  of  C4ISR  be  similarly  identified  (e.g., 
AWACS  deployment  impacting  prices  of  drugs  from  Colombia)? 


Key  Issues  and  Questions: 

1 .  The  nature  of  the  world,  the  threat,  has  changed  drastically  since  the  end  of  the  cold 
war.  The  distribution  of  risk  of  warfare  is  quite  different,  with  more  uncertainty 
about  the  probabilities  of  specific  threats.  There  is  a  need  to  collect  data  and  develop 
the  distribution  of  events  on  the  current  and  future  spectra  of  conflict. 

2.  The  Naval  Postgraduate  School  study  identified  significant  potential  savings  on  oil 
prices  by  early  deployment  of  naval  forces.  It  is  possible  that  early  deployment  of 
C4ISR  systems  could  provide  similar  benefits. 
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Analytical  Techniques  and  Tools: 

Key  Issues  and  Questions  (canted) 


3.  The  less  an  event  looks  like  war,  the  less  able  we  are  to  articulate  its 
governing  processes.  C4ISR  in  PE  requires  drawing  in  political 
constraints/considerations  to  a  degree  greater  than  for  other  points  on  the 
spectrum.  This  suggests  peculiar  data  requirements  for  military/C4ISR 
commanders  in  PE  settings. 

4.  For  PE  events,  the  cyclical  process  (define  MoM,  measure  sensitivity,  collect 
data,  evaluate  data,  report)  needs  to  also  capture  critical  C4ISR  data  and 
effects. 
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Analytical  Techniques  and  Tools: 

Recommendations 


Develop  broadly-accepted  TOR  for  PE;  definitions,  objectives,  goals, 

distinguishing  relationships  from 

OOTWtoMTW 


•  Develop  CSOP  for  PE 


•  Develop  MoMs  and  data  collection  checklists  for  PE 

Expand  JULLS  formats  and  databases  to  include  C4ISR  data 


Recommendations: 

Recognizing  PE  as  an  important  mission  area,  with  attendant  principles  and 
distinguishing  relationships,  will  assist  in  identifying  the  peculiar  tasks  of  PE  and 
gamering  funding  for  their  successful  undertaking. 

Common  Standard  Operating  Procedures  must  account  for  the  many  differences 
(historical,  cultural,  financial,  educational,  etc.)  that  exist  in  the  many  countries  that 
we  engage  dunng  PE  operations.  “One  size  fits  all”  is  not  the  answer.  We  need  to 
KNOW  our  PE  targets”  much  better  and  tailor  each  engagement  to  that  particular 
country.  Specific  C4ISR  exchanges  are  good  candidate  elements  for  tailored  PE. 

Expansion  of  the  Joint  Uniform  Lessons  Learned  database  to  include  C4ISR  data  and 
information  will  help  long  term  analysis  of  C4ISR  issues. 
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Analytical  Techniques  and  Tools: 

Recommendations  ( cont  ) 

•  study  the  impact  of  deploying  C4ISR  in  peacetime 

—  Assess  utility  of  C4ISR  deployment  on  its  own 

—  Assess  how  early  C4ISR  deployment  facilitates  early  combat  power  by 
follow-on  forces 


Recommendations  (conCd): 

The  results  of  such  a  study,  successfully  completed,  should  help  determine  the  value 
added  by  C4ISR  systems.  The  study  should  also  address  how  a  robust  C4ISR  exchange 
program  helps  build  coalitions  and  therefore  serves  as  a  significant  deterrent. 
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Analytical  Techniques  and  Tools: 

Assessment  of  Current  State^of-the-Practice 


Assessment  of  Current  State-of-the-Practice: 

Scenarios  represent  the  only  good  news  on  this  slide.  In  all  other  areas,  assessment  of  the 
current  state-of-the-practice  is  discouraging.  There  is  much  work  to  be  done  to  develop 
measures  of  effectiveness  and  tools  and  to  collect  data  relevant  to  proper  analysis  of 
C4ISR  issues  related  to  PE. 
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KEY  ISSUES 


Analytical  Techniques  and  Tools: 

Summary 


•  Since  the  cold  war,  the  spectrum  of  potential  conflict  has  broadened.  The 
probability  of  occurrence  of  events  on  this  spectrum  may  be  differently 
distributed.  Data  collection  and  analysis  of  that  issue  should  follow. 

•  The  less  an  event  looks  like  war,  the  less  able  we  are  to  articulate  its  governing 
processes.  C4ISR  in  PE  requires  drawing  in  political  constraints/ 
considerations  to  a  degree  greater  than  for  other  points  on  the  spectrum.  This 
suggests  peculiar  data  requirements  for  military/  C4ISR  commanders  in  PE 
settings. 

•  For  PE  events,  the  cyclical  process  (deHne  MoM,  measure  sensitivity,  collect 
data,  evaluate  data,  report)  needs  to  also  capture  critical  C4ISR  data  and 
effects. 


RECOMMENDATIONS 

•  Develop  broadly-accepted  TOR  for  PE:  definitions,  objectives,  goals, 
distinguishing  relationships  from  OOTW  to  MTW 

•  Develop  CSOP  for  PE 

•  Develop  MoMs  and  data  collection  checklists  for  PE 

•  Expand  JULLS  formats  and  databases  to  include  C4ISR  data 

•  Study  the  impact  of  deploying  C4ISR  in  peacetime.  Assess  the  utility  of 
C4ISR  deployment  on  its  own.  Assess  how  early  C4ISR  deployment  facilitates 
early  combat  power  by  follow-on  forces. 


Summary: 


KEY  ISSUES 

•  The  nature  of  the  world,  and  the  threat,  has  changed  since  the  end  of  the  Cold 
War.  The  distribution  of  risk  of  warfare  is  quite  different,  with  more 
uncertainty  about  the  probabilities  of  specific  threats.  There  is  a  need  to 
collect  data  and  develop  the  distribution  of  events  on  the  current  and  future 
spectra  of  conflict. 

RECOMMENDATIONS 

•  Recognizing  PE  as  an  important  mission  area,  with  attendant  principles  and 
distinguishing  relationships,  will  assist  in  identifying  the  peculiar  tasks  of  PE, 
and  gamering  funding  for  their  successful  undertaking. 

•  Common  Standard  Operating  Procedures  must  account  for  the  many 
differences  (historical,  cultural,  financial,  educational,  etc.)  that  exist  in  the 
many  countries  that  we  engage  during  PE  operations.  “One  size  fits  all”  is  not 
the  answer.  We  need  to  KNOW  our  PE  “targets”  much  better,  and  tailor  each 
engagement  to  that  peirticular  country.  Specific  C4ISR  exchanges  are  good 
candidate  elements  for  tailored  PE. 

•  Expansion  of  the  Joint  Uniform  Lessons  Learned  (JULLS)  database  to  include 
C4ISR  data  and  information  will  help  long  term  analysis  of  C4ISR  issues. 
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Analyzing  C4ISR  for  2010 
Synthesis  Group 
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Synthesis  Group 

Synthesis  Team 

• 

Major  Theater  Warfare  (MTW) 

• 

Infrastructure  Assurance 

“  Denny  Baer,  Logicon 

(lA) 

~  Jerry  Kotchka,  Boeing 

—  Stuart  Starr, 

• 

Smaller  Scale  Contingencies 

MITRE 

(SSC) 

• 

Information  Architectures 

-  Dave  Alberts,  OASD(C3I) 

—  Larry  Wiener,  DoN 

• 

Operations  Other  Than  War 

• 

Analytical  Techniques  and 

(OOTW) 

Tools 

-  Russ  Richards,  MITRE 

—  Denis  Clements, 

• 

Peacetime  Engagement 

GRCI 

-  Clay  Thomas,  USAF 

-  Jay  Kistler,  USN 

—  Bill  Kemple,  NPS 

—  Steve  Myer,  USA 

• 

Floaters 

-  Bob  Eberth,  USNR 

—  Pat  Peterson,  USN 

Synthesis  Team: 


The  Synthesis  Team  was  created  to  develop  a  holistic  view  of  the  deliberations  of  all  of 
the  Working  Groups  participating  in  the  Workshop.  Consistent  with  that  goal,  a  team 
was  selected  that  included  representation  from  OSD,  each  of  the  Services,  industry, 
FFRDCs  and  academia.  Selected  members  of  the  Synthesis  Team  actively  participated  in 
each  of  the  Working  Groups  (see  the  Viewgraph  for  the  specific  assignments).  The 
Synthesis  Team  convened  periodically  during  the  course  of  the  Workshop  to  discuss  and 
compare  the  evolving  findings  and  recommendations  of  each  Working  Group.  Drawing 
on  those  deliberations,  the  Synthesis  Team  developed  some  broad  observations  about  the 
nature  of  the  problem  and  formulated  a  set  of  cross-cutting  findings  and 
recommendations.  These  products  are  presented  and  discussed  below. 
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Synthesis  Group 

Broad  Observations 


•  There  is  broad  variability  in  the  “maturity”  of  the  mission  areas;  i.e.,  most 
mature  to  least  mature: 

-MTW 

-ssc 

-OOTW 

—  Peacetime  Engagement 
—  Infrastructure  Assurance 

•  Selected  mission  areas  (e.g.,  OOTW)  subsume  sub-missions  that  are 
qualitatively  different  (e.g.,  HATDR,  PK/PE) 


Broad  Observations: 

As  a  consequence  of  monitoring  the  deliberations  of  the  individual  Mission  Oriented 
Working  Groups  it  became  clear  that  there  was  broad  variability  in  the  maturity  of  the 
individual  mission  areas  (e.g.,  level  of  domain  knowledge;  available  tools  and  data; 
experience  in  analyzing  the  C4ISR  systems  and  processes  associated  with  those 
missions).  Based  on  the  perceptions  of  the  Synthesis  Team,  it  was  concluded  that  one 
could  rank  order  the  missions  on  a  qualitative  “maturity  scale”  as  follows:  Major  Theater 
War,  Smaller  Scale  Contingency,  Operations  Other  Than  War,  Peacetime  Engagement 
and  Infrastructure  Assurance. 

The  Synthesis  Team  observed  further  that  many  of  these  mission  areas  subsume 
qualitatively  different  sub-missions.  For  example.  Operations  Other  Than  War  subsumes 
Humanitarian  Assistance,  Disaster  Relief,  Peace  Keeping  and  Peace  Enforcement.  These 
sub-missions  are  so  different  in  their  nature  and  maturity  that  it  can  be  misleading  to 
make  overarching  statements  about  the  overall  mission  area. 
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Synthesis  Group 

Major  Issues 


Mission  Area 

Major  Issues 

MTW 

Few  accepted  data  sources;  Modeling  Red  C4ISR;  Existing  data 
does  not  map  to  DPG  scenarios 

SSC 

MoMs  need  sufficient  pre-definition 

OOTW 

Data;  Softness  of  MoMs  (public  opinion);  OOTWs  are  not  all  the 
same 

Peacetime  Engagement 

Definition 

Infrastructure 

Assurance 

Data  (much  of  it  is  Blue  data);  Scope  of  problem/ability  to 
decompose  meaningfully;  Higher  level  MoM^  Dealing  with  the 
interactive  nature  of  the  problem 

Info  Arch 

How  does  one  score  an  architecture  and  communicate  the  results 
appropriately? 

Tools  &  Tech 

Cognitive  modeling;  Broad  array  of  missions;  Size  of  the  trade 
space;  Representation  of  Red  data 

Major  Issues: 

As  the  Synthesis  Team  monitored  the  deliberations  of  the  individual  Working  Groups  it 
identified  several  major  issues  that  were  raised  by  those  Working  Groups.  The 
associated  viewgraph  briefly  summarizes  those  major  issues.  Note  that  although  there 
are  considerable  differences  in  those  issues,  several  cross-cutting  issues  are  apparent. 
These  cross-cutting  issues  involve  difficulties  associated  with  data  and  measures  of  merit 
(MoMs).  These  specific  cross-cutting  issues  will  be  highlighted  in  the  findings  and 
recommendations  that  follow. 
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Synthesis  Group 

Aggregate  Assessment 


Aggregate  Assessment: 

In  his  plenary  presentation,  Stuart  Starr  introduced  a  template  that  was  developed  in  the 
forthcoming  NATO  Code  of  Best  Practice  (COBP)  for  Assessing  Command  and 
Control.  [Note:  for  information  purposes,  a  brief  svunmary  of  the  highlights  of  the 
NATO  COBP  is  included  as  Annex  A].  That  template  identified  the  life  cycle  of  the  C2 
assessment  process,  emphasizing  the  key  steps  that  are  followed  by  leading  practitioners 
of  C2  assessment.  Each  of  the  Working  Groups  used  this  template  to  assess  the  current 
state  of  the  health  of  C2  assessment  for  their  mission  area.  The  associated  viewgraph 
provides  an  aggregate  assessment,  qualitatively  integrated  across  the  results  of  the 
individual  Working  Group  assessments.  It  can  be  seen  that  two  of  the  critical  steps  (i.e., 
data,  tools)  were  assessed  as  being  poor  (“Red”),  five  of  the  steps  (i.e.,  problem 
structuring,  treatment  of  human  factors/organization,  measures  of  merit,  treatment  of  risk 
and  uncertainty,  report  documentation  and  availability)  were  assessed  as  being 
significantly  deficient  (“orange”),  and  one  step  (i.e.,  scenario  generation)  was  assessed 
as  being  fair  (“yellow”).  It  is  notable  that,  on  aggregate,  none  of  the  steps  were  assessed 
as  being  good  (‘green”). 

These  observations  suggest  that  there  is  widespread  concern  about  the  conununity’s 
ability  to  adhere  to  the  processes  in  a  strawman  COBP.  These  results  led  the  Synthesis 
Group  to  highlight  findings  and  recommendations  in  five  areas:  COBP,  three  key 
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elements  of  the  candidate  COBP  and  the  overall  relationship  between  the  producer  and 
consumer  of  C2  analyses.  These  findings  and  recommendations  are  summarized  below. 
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Synthesis  Group 

Code  of  Best  Practices  (COBP): 
Findings 


•  Findings 

—  A  community-endorsed  COBP  is  needed  to  provide  a  sound 
foundation  for 

•  structuring, 

•  addressing  the  C2  assessment  problem,  in  all  its  dimensions 


Code  of  Best  Practices  (COBP)  -  Findings: 

The  Synthesis  Working  Group  concluded  that  the  NATO  COBP  represented  an 
interesting  initial  step  towards  developing  a  sound  foundation  for  C2  assessment. 
However,  that  product  was  restricted  in  its  scope  to  assessing  C2  in  the  context  of 
conventional  warfare.  As  this  Workshop  revealed,  the  problem  of  analyzing  C4ISR  for 
2010  will  involve  a  host  of  additional  mission  areas,  which  pose  unique  problems  for  the 
C2  assessment  community.  Thus,  an  expanded,  community-endorsed  COBP  is  needed  to 
support  the  C2  assessment  problem,  in  all  its  dimensions. 
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Synthesis  Group 

Code  of  Best  Practices  (COBP): 
Recommendations 

Recommendations: 

•  An  assessment  should  be  made  of  the  NATO  COBP  to  see  if  it  meets 
the  needs  of  the  US  C2  assessment  community  (Action:  MORS) 

•  If  the  assessment  endorses  the  COBP,  it  should  be 

broadly  disseminated  to  the  C2  assessment  community 
(Action:  MORS) 

“  employed  in  defense  universities  to  educate  young  analysts 
(Action:  NPS,AFIT) 

•  Efforts  should  be  supported  to  extend  the  NATO  COBP  to  address 
New  World  Disorder  issues  (Action:  SAS  Panel-015) 


Code  of  Best  Practices  (COBP)  -  Recommendations: 

Although  the  preliminary  review  of  the  NATO  COBP  was  promising,  it  is  recommended 
that  a  more  thorough,  in-depth  assessment  of  the  product  should  be  undertaken  to  see  if 
it  meets  (at  least  some  of)  the  needs  of  the  US  C2  analysis  community.  MORS  is  in  the 
best  position  of  any  US  organization  to  organize  and  execute  such  an  assessment. 

If  such  an  assessment  is  undertaken  and  it  endorses  the  NATO  COBP  as  a  preliminary 
product,  the  following  steps  should  be  pursued.  First,  the  product  should  be  disseminated 
broadly  to  the  C2  assessment  community,  along  with  any  caveats  that  emerged  through 
the  assessment  process.  MORS  has  an  efficient  distribution  network  for  making  such 
information  broadly  available.  Second,  consideration  should  be  given  to  employing  the 
product  in  defense  universities,  such  as  the  Naval  Postgraduate  School  (NPS)  and  the 
Air  Force  Institute  of  Technology  (AFTT),  to  educate  young  analysts. 

In  view  of  the  limited  scope  of  the  NATO  COBP,  efforts  should  be  supported  to  extend 
the  product  to  address  New  World  Disorder  issues.  NATO  is  forming  Studies,  Analysis 
and  Simulation  (SAS)  Panel-015,  to  explore  the  feasibility  of  extending  the  initial  COBP 
to  deal  with  issues  associated  with  OOTW  and  Information  Operations.  Those  efforts 
should  be  supported  by  MORS  (e.g.,  by  providing:  representatives  to  the  deliberations; 
appropriate  briefings/case  studies  to  the  participants;  a  peer-review  of  the  panel’s 
product)  to  ensure  that  the  resulting  efforts  are  of  greatest  utility  to  the  US  C2 
assessment  community. 
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Synthesis  Group 

Measures  of  Merit  (MoMs):  Findings 

•  Findings 

MORS’  concept  of  a  hierarchy  of  MoMs  appears  valid 
—  However, 

•  the  hierarchy  must  be  extended  to  deal  with  the  needs  of  New 
World  Disorder  missions 

•  MoMs  are  lacking  to  support  the  effective  assessment, 
comparison  of  architectural  options 


Measures  of  Merit  -  Findings: 

Over  the  last  thirteen  years,  MORS  has  conducted  several  workshops  on  the  issue  of 
developing  Measures  of  Merit  (MoMs)  for  command  and  control  assessments  [Ref.  1, 
2].  Those  workshops  proposed  the  concept  of  a  hierarchy  of  MoMs  that  would  range 
from  measures  of  system  performance  through  measures  of  mission  effectiveness.  [Note: 
see  Figure  3  in  the  appended  paper]  .The  various  mission-oriented  working  groups  at  this 
workshop  appeared  to  support  this  conceptual  approach  to  the  problem.  However,  it  was 
concluded  that  the  hierarchy  must  be  extended  to  deal  with  the  needs  of  New  World 
Disorder  missions. 

The  Architecture  Working  Group  also  observed  that  it  lacked  MoMs  to  support  the 
effective  assessment  and  comparison  of  architectural  options.  In  particular,  it  called  for 
MoMs  that  would  relate  system  architectural  characteristics  to  operational  outcomes. 
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Synthesis  Group 

Measures  of  Merit  (MoMs):  Recommendations 


•  Recommendations 

Formulate  upper  levels  of  the  MoM  hierarchy  to  reflect  effectiveness 
of  C4ISR  vi'ith  respect  to 

•  political-institutional  stability  (e.g.,  OOTW,  SSC) 

•  public  conHdence  in  institutions  (e.g.,  lA) 

Explore  options  to  develop  MoMs  that  can  be  used  to  assess  and 
compare  architectural  options  (e.g.,  relate  system  architectural 
characteristics  to  operational  outcomes) 

(Action:  OSD  host  key  decision  makers;  MORS  facilitate) 


Measures  of  Merit  —  Recommendations: 

To  respond  to  the  needs  of  New  World  Disorder  missions,  efforts  are  needed  to 
formulate  the  upper  levels  of  the  MoM  hierarchy.  For  example,  in  the  mission  areas  of 
Smaller  Scale  Contingencies  and  Operations  Other  Than  War,  measures  are  needed  to 
characterize  political-institutional-social  stability  (e.g.,  are  the  bodies  of  government 
functioning  effectively;  are  educational  institutions  operative;  are  children  playing  on 
soccer  fields).  Similarly,  in  the  area  of  Infrastructure  Assurance,  there  is  a  need  for 
MoMs  that  reflect  public  confidence  in  key  infrastructures  (e.g.,  confidence  in  the 
finance  and  banking  sector  as  measured  by  the  value  of  the  Dow  Jones  Industrial 
Average  and  the  value  of  the  dollar  versus  the  Euro).  These  needs  suggest  the  need  for 
MORS  to  convene  a  workshop  on  expanded  C4ISR  MoMs  for  New  World  Disorder 
missions. 

To  satisfy  the  needs  of  the  Architectural  community,  steps  should  be  taken  to  explore 
options  to  develop  MoMs  that  can  be  used  to  assess  and  compare  architectural  options. 
One  possible  course  of  action  would  call  for  OSD  to  convene  an  Integrated  Process 
Team  (IPT)  of  stakeholder  organizations  to  address  this  issue.  MORS  could  provide  the 
facilitators  to  conduct  such  an  event. 
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Synthesis  Group 

Data:  Findings 

•  Findings 

—  Data  issues  were  highlighted  by  each  Working  Group 

—  Specific  concerns  varied  by  Group 

•  Dealing  with  data  appropriate  to  future  operations  (e.g.,  MTW) 

•  Acquiring  needed  Blue  system  data  (e.g.,  lA) 

•  Characterizing  Red  C4ISR,  infrastructure  (e.g.,  MTW,  lA) 

•  Gaining  access  to  data  in  a  timely  fashion  (All) 

•  Collecting  and  making  better  use  of  data  emerging  from 
experimentation  (e.g.,  MTW,  SSC,  OOTW) 

•  “...there  is  no  V&C  worthy  of  the  name”  (Eberth) 


Data  -  Findings: 

A  decade  ago,  at  a  MORS  Workshop  on  Simulation  Technology  1997  (SIMTECH  97), 
Dr.  Walt  LaBerge  observed  that  “without  data  we  are  nothing”  [Ref.  3].  That  thought 
clearly  prevailed  in  this  workshop  and  data  issues  were  highlighted  by  each  Working 
Group.  The  specific  concerns  varied  by  Working  Group  and  included  the  following 
challenges:  acquiring  the  data  appropriate  to  some  currently  ill-defined  future  military 
operation;  acquiring  the  Blue  system  data  needed  to  perform  vulnerability  assessments 
of  critical  infrastmctures;  characterizing  Red’s  C4ISR  systems  and  critical 
infrastructures;  gaining  access  to  needed  data  in  a  timely  fashion;  collecting  and  making 
better  use  of  data  emerging  from  Service  and  Joint  Advanced  War  fighting  Experiments. 
One  of  the  members  of  the  Synthesis  Working  Group  also  observed  that  there  is  “...  no 
validation  and  certification  worthy  of  the  name.” 
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Synthesis  Group 

Data:  Recommendations 

Establish  an  institutional  mechanism  for  acquiring,  W&Cing, 
transforming,  storing  and  making  accessible  key  data 

““  Draw  on  the  capabilities  and  resources  of  key  organizations  (e.g., 
DSC,  PA&E  (e.g.,  JDS),  DMSO  (e.g.,  MSOSA),  J6  (e.g.,  JDUCS, 
NETWARS))  (Action:  OASD(C3I)) 


Data  -  Recommendations: 

The  depth  and  breadth  of  these  data  concerns  are  such  that  a  new  institutional 
mechanism  is  needed  to  deal  with  the  totality  of  the  problem.  This  mechanism  should 
deal  with  the  full  life  cycle  of  the  data  problem,  to  include  data  acquisition,  verification, 
validation  and  certification  (VV&C),  transformation  (into  a  form  that  is  useful  to  the  C2 
assessment  community),  storage  and  access.  Currently,  there  are  a  number  of 
orgamzations  that  are  involved  with  significant  facets  of  the  problem.  These  include 
OSD’s  C4ISR  Decision  Support  Center  (DSC),  PA&E’s  Joint  Data  System  (JDS), 
Defense  Modeling  and  Simulation  Office  (DMSO)  Modeling  and  Simulation 
Operational  Support  Activities  (MSOSA)  and  J6’s  Joint  Defense  Information 
Infrastructure  Control  System  (JDIICS)  and  NETWARS  activities.  OASD(C3I)  should 
take  the  lead  in  forging  these  fragmented  efforts  into  a  complete,  mutually  reinforcing 
solution  to  the  total  problem. 
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Synthesis  Group 

Tools  and  Their  Application:  Findings 


*  Findings 

“  There  is  no  single  tool  (or  class  of  tools)  that  can  satisfy  C2 
assessment  needs  adequately 


Tools  and  Their  Application  -  Findings: 

Many  of  the  speakers  at  the  workshop  focused  their  remarks  on  modeling  and  simulation 
and  its  potential  role  in  C2  assessment.  It  is  clear,  however,  that  no  single  class  of  tool 
(and  certainly  no  single  tool  within  a  class)  can  satisfy  C2  assessment  needs  adequately. 
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Synthesis  Group 

Tools  and  Their  Application:  Recommendations 

*  Recommendations 

Encourage  the  use  of  a  broad  mix  of  tools  that  are  well  suited  to  the 
critical  issues  of  interest;  e.g., 

•  analyses,  based  on  basic  principles 

•  structured  expert  elicitation  tools 

•  creative  applications  of  “New  Sciences”  to  deal  with  Complex 
Adaptive  Systems 

•  multi-functional  tools  (e.g.,  support  to  analysis  and  rehearsal) 

~  Assiduously  pursue  efforts  to  perform  responsible  levels  of  VV&A 
(Action:  MORS) 

Apply  these  tools  iteratively  (e.g.,  generalization  of  the  Model-Test- 
Model  paradigm) 


Tools  and  Their  Application  -  Recommendations: 

There  is  a  broad  spectrum  of  tools  that  a  C2  analyst  can  chose  from,  that  vary  with 
respect  to  the  time  to  create  and  apply  them,  the  cost  to  create  and  apply  them  and  their 
credibility.  [Note:  see  Figure  4  in  the  appended  paper].  It  is  important  that  C2  assessors 
be  aware  that  this  broad  mix  of  tools  exists  (beyond  the  “usual  suspects”  of  M&S)  and 
be  conversant  about  their  strengths  and  weaknesses.  In  particular,  where  appropriate,  C2 
assessments  should  take  better  advantage  of  analyses,  based  on  basic  operations  research 
and  physical  principles;  structured  expert  elicitation  tools  (e.g.,  group  ware);  applications 
of  the  “New  Sciences”  to  deal  with  Complex  Adaptive  Systems;  and  multifonctional 
tools  (e.g.,  tools  that  can  support  analyses  and  mission  rehearsal).  In  all  instances, 
assessors  should  assiduously  pursue  efforts  to  perform  responsible  levels  of  verification, 
validation  and  accreditation  (VV«&;A)  for  the  tools  selected.  MORS  should  lead  the  way 
in  determining  what  is  a  “responsible  level  of  VV&A.”  [Note:  it  is  anticipated  that 
SIMVAL  99  will  shed  light  on  this  issue]. 

Because  of  the  strengths  and  weaknesses  of  these  tools,  it  is  important  to  select  and 
orchestrate  a  complementary  set  of  tools.  In  the  past,  this  concept  has  been  limited  to  the 
Model-Test-Model  paradigm.  However,  it  would  be  appropriate  to  generalize  that 
concept  to  incorporate  a  broader  set  of  tool  types.  For  example,  it  might  be  desirable  to 
consider  a  “expert  elicitation-real  world  experience-test-model”  paradigm,  that  expands 
the  set  of  tools  that  are  orchestrated. 
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Synthesis  Group 

Provider-Comiimer  Relations:  Findings 

•  Findings 

—  Providers  are  failing  to  make  their  analyses  transparent, 
understandable 

—  Consumers  are  frequently  not  ‘‘educated  customers” 


Provider-Consumer  Relations  -  Findings: 

Many  of  the  Working  Groups  observed  that  relations  between  the  providers  and 
consumers  of  C2  assessment  are  strained.  They  noted  that  many  providers,  by  relying  on 
complex,  opaque  M&S  are  failing  to  make  their  analyses  transparent  and  understandable 
(e.g.,  “Why  did  I  come  to  that  conclusion?  My  model  told  me!”).  Similarly,  consumers 
are  frequently  not  “educated  customers.”  For  example,  they  fail  to  articulate  issues  in  a 
way  that  is  amenable  to  responsible  C2  assessment,  or  they  fail  to  give  the  provider 
adequate  resources  to  either  create  the  needed  tools  or  to  perform  the  assessment. 
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Synthesis  Group 

Provider-Consumer  Relations:  Recommendations 

•  Recommendations 

”  There  should  be  a  “contract”  between  providers  and  consumers, 
recognizing  the  mutual  responsibilities  of  both  sides 

A  COBP  should  be  developed  (perhaps  in  the  form  of  a  check  list)  to 
help  educate  consumers  about  the  attributes  of  sound  C4ISR 
assessment 
(Action:  MORS) 


Provider-Consumer  Relations  -  Reconunendations: 

This  issue  was  recognized  and  addressed  at  the  MORS  Mini-Symposium  on  “Quick 
Reaction  Analysis  Requirements  and  Methodologies  (QRAM)”  [Ref.  4],  The  Synthesis 
Working  Group  at  that  workshop  recognized  that  there  should  be  a  “contract”  between 
providers  and  consumers,  recognizing  the  mutual  responsibilities  of  both  sides.  It  is 
recommended  that  the  draft  “contract”  be  updated  and  refined  to  reflect  the  lessons 
learned  from  this  workshop. 

As  a  further  step,  a  COBP  should  be  developed  (perhaps  in  the  form  of  a  checklist)  to 
help  educate  consumers  about  the  attributes  of  sound  C4ISR  assessment.  This  “check 
list”  could  be  patterned  after  the  COBP  that  the  AIAA  Information  &  C2  System 
Technical  Comnuttee  is  developing  for  OSD  on  C2  experimentation. 
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Synthesis  Group 

Summary 

•  Good  News 

—  We  are  beginning  to  understand 

•  the  importance  of  C4ISR 

•  the  practices  that  should  be  followed  to  perform  credible  C4ISR 
analyses 

•  ...  and  what  we  don’t  understand! 

•  Bad  News 

—  Currently,  we  generally  do  not  do  a  satisfying  job  in  performing  C4ISR 
analyses 

•  Worse  News 

—  With  the  arrival  of  the  New  World  Disorder,  the  C4ISR  assessment 
problem  is  getting  substantially  more  difticult 

•  Better  News 

—  If  the  recommendations  of  the  workshop  are  implemented,  it  will  provide 
the  foundation  for  the  C41SR  community  to  work  collaboratively  to  attack 
the  most  critical  of  our  shortfalls 


Summary: 

The  Synthesis  Group  discerned  news  on  C2  assessment  that  ran  the  gamut  from 
good  to  bad  to  worse  to  better. 

The  good  news  is  that  we  are  beginning  to  understand  a  great  deal  more  about  the 
C2  assessment  problem.  We  are  starting  to  recognize  the  importance  of  C4ISR  in 
the  context  of  the  many  missions  that  DoD  must  perform;  the  practices  that 
should  be  followed  to  perform  credible  C4ISR  analyses; ...  and  what  we  don’t  yet 
understand! 

The  bad  news  is  that  currently,  we  generally  do  not  do  a  satisfying  job  in 
performing  C4ISR  analyses.  Although  there  are  counter-examples  to  this 
statement,  it  is  recognized  that  most  studies  of  conventional  conflict  fail  to  live  up 
to  the  COBP  generated  by  NATO’s  SAS  Pzinel-002. 

The  worse  news  is  that  the  C2  assessment  problem  is  getting  substantially  more 
difficult.  We  are  confronted  with  a  host  of  New  World  Disorder  missions  and 
issues  for  which  we  lack  the  key  elements  of  effective  C2  analyses  (particularly 
relevant  data  and  tools).  In  addition,  these  issues  are  in  a  constant  state  of  flux, 
making  it  difficult  to  get  traction  on  the  problem.  One  of  the  most  dramatic 
examples  of  this  challenge  is  the  Infrastructure  Assurance  problem,  where  the 
infrastructures  and  the  threats  to  them  are  changing  dramatically. 
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Although  these  are  daunting  challenges,  there  is  some  better  news  that  is  implicit 
in  this  workshop  itself.  If  the  recommendations  of  the  workshop  are  implemented 
successfully,  it  will  have  provided  the  foundation  for  the  C4ISR  community  to 
work  collaboratively  to  attack  the  most  critical  of  our  shortfalls. 
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C4ISR  Mini-SymposiumAVorkshop 
Acronyms 


AFTT 

AFSA/SAAB 

AIAA 

ARES 

ASD 

ASD(C3I) 

AWACS 

BMDO 

C2 

C3 

C3I 

C3ffiW 

C4I 

C4ISR 

CAS 

CCIR 

CCRP 

CENTCOM 

CIP 

CMA 

COA 

COBP 

COMINT 

CONOPS 

COP 

COTS 

CSOP 

DCI 

DD 

DDR&E 


Air  Force  Institute  of  Technology 
Airforce  Studies  and  Analysis  Activity 
American  Institute  of  Astronautics  and  Aeronautics 
Regional  Exploratory  Simulation 
Assistant  Secretary  of  Defense 

Assistant  Secretary  of  Defense  for  Command,  Control,  Communications 
and  Intelligence 

Airborne  Warning  and  Control  Systems 

Ballistic  Missile  Defense  Office 

Command  and  Control 

Command,  Control  and  Communications 

Command,  Control,  Communication  and  Intelligence 

Connnand,  Control,  Communications,  Intelligence  and  Electronic 

Warfare 

Command,  Control,  Communications,  Computers  and  hitelligence. 

Command,  Control,  Conmiunications,  Computers,  Intelligence, 

Surveillance  and  Reconnaissance 

Complex  Adaptive  Systems 

Commander’s  Critical  Information  Requirement 

C2  Research  Program 

[US]  Central  Command 

Critical  Infrastructure  Protection 

C4ISR  Mission  Assessment 

Course  of  Action. 

Code  of  Best  Practices 
Communications  Intelligence 
Concept  of  Operations 
Common  Operational  Picture 
Commercial-Off-The-Shelf 
Current  State  of  Practice 

1.  Director  of  Central  Intelligence;  2.  Dual  Channel  Interchange;  3. 
Document  Change  Instruction 
Variation  of  DoD 

Director  Defense  Research  and  Evaluation 
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DepSECDEFA^CJCS 

Deputy  Secretary  of  DefenseA^ice  Chairman,  Joint  Chiefs  of  Staff 

DM 

Decision  Maker 

DMSO 

Defense  Modeling  and  Simulation  Office 

DoD 

Department  of  Defense 

DoN 

Department  of  the  Navy 

DPG 

Defense  Planning  Guidance 

DR/HA 

Disaster  Relief/Humanitarian  Assistance 

DSC 

Decision  Support  Center 

DSTF 

Decision  Support  Task  Force 

ELINT 

Electronic  Intelligence 

PER 

Force  Exchange  Ratio 

FFRDC 

Federally  Funded  Research  and  Development  Center 

GO 

General  Officer 

GPS 

1.  Global  Positioning  System;  2.  Global  Protection  System 

GRCI 

General  Research  Corporation  Inc 

HA/DR 

Higher  Authority/Deployment  Review 

HEAT 

Headquarters  Effectiveness  Assessment  Tool 

HUMINT 

Human  Intelligence 

lA 

Infrastmcture  Assurance 

ICBM 

Inter-Continental  Ballistic  Missile 

IFFN 

Identify  Friend  or  Foe  Network 

MINT 

Imagery  Intelligence 

lO 

Information  Operations 

IPB 

Intelligence  Preparation  of  the  Battlefield 

IPS 

Integrated  Program  Summary 

IPT 

Integrated  Process  Team 

IR 

Infrared 

ISR 

Intelligence,  Surveillance  and  Reconnaissance 

J-2 

Joint  Staff  Section  2  (fritelligence) 

J-6 

Joint  Staff  Section  6  (Command,  Control,  Communications  and 
Computers) 

J-8 

Joint  Staff  Section  8 

JAMIP 

Joint  Analytic  Model  Improvement  Program 

JBC 

Joint  C4ISR  Battle  Center 

JCAPS 

Joint  C4SI  Architecture  Planning/Analysis  System 

JDHCS 

Joint  Defense  Information  Infrastructure  Control  System 

JDS 

Joint  Data  System 

JROC 

Joint  Requirements  Oversight  Council 
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JTF 

Joint  Task  Force 

JWARS 

Joint  Warfighting  System 

JWCA 

Joint  Warfighting  Capability  Assessment 

M&S 

Modeling  and  Simulation 

MAGTG 

Marine  Air  Ground  Task  Force 

MASINT 

Measurement  and  Signals  Intelligence 

MCES 

Modular  C2  Evaluation  Structure 

MOA 

Mission  Oriented  Approach 

MOEs 

Measure  of  Effectiveness 

MOFEs 

Measure  of  Force  Eflfectiveness 

MoME 

Measure  of  Mission  Effectiveness 

MoMs 

Measures  of  Merit 

MoPE 

Measure  of  Policy  Effectiveness 

MoPs 

Measure  of  Performance 

MORS 

Military  Operations  Research  Society 

MOPE 

Measure  of  Performance  Effectiveness 

MSOSA 

Modeling  and  Simulation  Operational  Support  Activities 

MTI 

Moving  Target  Indicator 

MTW 

Major  Theater  War 

NAI’s 

Named  Area’s  of  Interest 

NATO 

North  Atlantic  Treaty  Organization 

NEO 

Non-Combatant  Evacuation  Operation 

NGO 

Non  Governmental  Organization 

NPS 

Naval  Postgraduate  School 

NSS 

Naval  Simulation  System 

OOTW 

Operations  Other  Than  War 

OP 

Optical  Processing 

OPNAV 

Office  of  the  Chief  of  Naval  Operations 

OASD 

Office  of  the  Assistant  Secretary  of  Defense 

OSD 

Office  of  the  Secretary  of  Defense 

PA«&E 

Program  Analysis  and  Evaluation 

PACOM 

[United  States]  Pacific  Command 

PDD 

Presidential  Decision  Directive 

PE 

Peacetime  Engagement 

PED 

Processing,  Exploitation  and  Dissemination 

PK/PE 

Peace  Keeping/Peace  Enforcement 

PK/PE/PB 

Peace  Keeping/Peace  Enforcement/Peace  Building 

POM 

Program  Objective  Memorandum 
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PPBS 

Planning,  Programming,  Budgeting  System 

PPE 

President’s  Program  Element 

PSYOPS 

Psychological  Operations 

QDR 

Quadrennial  Defense  Review 

QRAM 

Quick  Response  Analysis  Methodologies 

Recce 

Reconnaissance 

RMA 

Revolutionary  in  Military  Affairs 

ROE 

Rules  of  Engagement 

ROI 

Return  on  Investment 

SAC 

Strategic  Air  Command 

SAS 

Panel  Studies,  Analysis  and  Simulations 

SBA 

Simulation  Based  Acquisition 

SCUDs 

Surface  to  Surface  Missile  System 

SECDEF 

Secretary  of  Defense 

SIGINT 

Signals  Intelligence 

SIMTECH 

Simulation  Technology 

SIMVAL 

Simulation  Validation 

SPAM 

Sensor  Platform  Allocation  Model 

SSC 

Smaller  Scale  Contingencies 

TASC 

The  Analytical  Sciences  Group 

TMD 

Theater  Missile  Defense 

TOR 

Terms  of  Reference 

TRAC-WSMR 

Training  and  Doctrine  Command  Analysis  Center/White  Sands  Missile 
Range 

UHF 

Ultra  High  Frequency 

UN 

United  Nations 

US 

United  States 

USA 

United  States  Army 

USACOM 

United  States  Atlantic  Command 

USFK 

United  States  Forces  Korea 

USN 

United  States  Navy 

v&c 

Validation  and  Certification 

v&v 

Verification  and  Validation 

VIC 

Vector  In  Commander 

W&A 

Verification,  Validation  and  Accreditation 

w&c 

Verification,  Validation  and  Certification 

WMD 

Weapons  of  Mass  Destruction 
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TERMS  OF  REFERENCE 
Analyzing  C4ISR  for  2010  Workshop 


Background 

The  Revolution  in  Military  Affairs  has  already  begun  to  transform  our  command  and  control 
procedures  and  architectures,  as  we  straggle  to  adapt  our  institutions  to  the  realities  of  the 
Information  Age.  Recent  seminal  defense  planning  documents,  e.g..  Joint  Vision  2010, 
Quadrennial  Defense  Review,  and  the  National  Defense  Panel  report,  have  highlighted  the  critical 
role  of  information  technology-based  C4ISR  for  future  US  forces.  A  broader  spectrum  of  threats 
and  operations,  a  smaller  more  capable  force,  and  an  information  technology-based  society 
require  greatly  improved  C4ISR  capabilities.  In  our  resource-constrained  environment,  however, 
decision  makers  face  tough  choices  in  determining  the  allocation  of  investment  between  C4ISR 
and  the  other  contributors  to  force  effectiveness  (e.g.,  platforms  and  weapons,  modernization, 
sustainment,  logistics).  The  analytical  community  must  be  able  to  provide  cogent  analyses  to 
support  these  decisions.  What  has  been  missing  heretofore  is  the  ability  to  assess  effectively  the 
relative  contribution  of  C4ISR  to  force  effectiveness  vis-a-vis  other  factors.  Past  analyses  —  with 
their  component  metrics,  methodologies,  and  tools  —  have  not  been  adequate  for  this  new 
emphasis.  The  analytic  community  must  now  posture  itself  to  address  the  analysis  of  C4ISR  for 
2010  and  beyond. 

This  special  meeting,  comprising  analysts  and  C4ISR  subject  matter  experts,  will  concentrate  on 
the  improvement  of  analysis  of  C4ISR  for  a  US  force  and  doctrine  heavily  reliant  on  information 
technologies.  To  achieve  this  end,  participants  will  enumerate  potential  decision  maker  issues  and 
deHne  the  appropriate  analytic  metrics,  methodologies,  and  tools  for  C4ISR  analysis.  These 
issues  will  be  addressed  by  working  groups  covering  the  spectrum  of  military  operations,  as  well 
as  information  infrastructures  and  leading-edge  analytical  techniques. 

Goals  and  Objectives 

finals.  This  workshop  will  afford  the  military  OR  community  an  opportunity  to  achieve  the 
following  goals:  (1)  share  information  on  the  current  state  of  C4ISR  analysis;  (2)  determine  the 
appropriate  metrics  and  methodologies  for  analysis  of  C4ISR  for  2010;  and  (3)  define  key  areas 
of  improvement  for  the  application  of  analysis  in  support  of  decision  making  on  C4ISR 
investments  and  warfighting  utility.  The  results  will  provide  insights  into  how  to  advance  the 
ability  of  the  OR  community  to  conduct  analysis  of  C4ISR  in  support  of  Joint  Vision  2010  and 
beyond. 

Objectives.  The  objectives  are  to: 

•  Enumerate  the  relevant  issues  pertaining  to  the  analysis  of  C4ISR  in  2010 

•  Identify  metrics  that  are  sensitive  to  the  effects  of  C4ISR  on  force-level  effectiveness 

•  Assess  methodologies  to  analyze  and  quantify  the  effectiveness  of  C4ISR 

•  Evaluate  appropriate  tools  to  measure  the  benefit  of  C4ISR 

•  Discuss  the  requirements  for  and  employment  of  advanced  tools,  methods,  and  research. 
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Approach 


In  order  to  achieve  these  goals  and  objectives,  the  subject  of  C4ISR  analysis  will  be  exanndned  in 
a  number  of  different  contexts.  A  working  group  will  examine  C4ISR  analysis  within  each 
contextual  framework.  Working  groups  will  be  asked  to  address  some  baseline  issues  (as  a 
starting  point)  and  provide  specific  results.  The  working  group  contexts,  baseline  issues,  and 
desired  results  are  detailed  below. 

Working  Groups.  The  workshop  attendees  will  be  organized  into  working  groups  (18-20 
members  each)  to  examine  the  analysis  of  C4ISR  within  these  specific  focus  areas: 

•  Major  Theater  War 

•  Smaller  Scale  Contingencies 

•  Operations  Other  Than  War 

•  Infrastructure  Assurance 

•  Peacetime  Engagement 

•  Information  Architectures 

•  Analytical  Techniques  and  Tools 

•  Synthesis 

A.  Major  Theater  War.  Co-chairs:  Dr.  Mark  Youngren  and  Chuck  Taylor.  This  working 
group  will  focus  on  C4ISR  within  the  context  of  large-scale,  joint  and  combined  operations 
against  a  technologically  sophisticated  adversary.  These  highly  complex  operations  will  require 
fusion  of  all-source  intelligence,  seamless  integration  of  sensors,  platforms,  and  command 
organizations,  and  intense  logistic  support  to  allow  a  greater  number  of  operational  tasks  to  be 
accomplished  faster.  An  important  consideration  for  this  group  is  ascertaining  our  ability  to 
determine  the  relative  contributions  of  C4ISR  and  platforms/weapons  to  force  effectiveness. 

B.  Smaller  Scale  Contingencies.  Chair:  Dick  Hayes.  This  working  group  will  focus  on  C4ISR 
in  the  context  of  smaller  scale  operations,  including  "show  of  force”,  crisis  response,  raids, 
opposed  (or  potentially  opposed)  noncombatant  evacuations,  sanction  enforcement,  special 
operations,  and  counterterrorism.  These  operations  may  require  covert  Intelligence-Surveillance- 
Reconnaissance  (ISR)  and  communications,  flexible  contingency  planning,  command  and  control 
responsive  to  national  as  well  as  tactic2d  concerns,  and  highly  portable,  reconfigurable,  and  secure 
joint  support  systems.  They  may  also  involve  interagency  or  coalition  issues. 

C.  Operations  Other  Than  War.  Co-chairs:  Dr.  Cy  Staniec,  John  Furman  and  Terry  Prosser. 
This  working  group  will  focus  on  C4ISR  within  the  context  of  OOTW.  In  order  to  bolster 
regional  stability,  US  forces  are  increasingly  likely  to  be  involved  in  disaster  relief,  humanitarian 
assistance,  peacekeeping  and  peace  enforcement,  and  similar  non-traditional  military  missions.  In 
these  settings,  coordination  with  non-DoD  entities  (e.g..  State  Department),  non-governmental 
organizations  (e.g..  International  Red  Cross),  and  host  nation  governments  will  entail  unique 
C4ISR  requirements. 

D.  Infrastructure  Assurance.  Co-chairs:  Thomas  Bozek  and  Frank  Ruggeri.  This  working 
group  will  focus  on  defense  of  key  elements  of  national  infrastructure.  DoD  is  dependent  on  a 
complex  framework  of  networks  and  systems  to  provide  the  continual  flow  of  essential  goods  and 
services,  including  information,  electrical  power,  finance  and  banking,  transportation,  fuel 
distribution,  health  care,  and  emergency  services.  The  military  infrastructure  is  characterized  by 
increasingly  interdependent  mix  of  defense  and  commercial  services;  reliance  on  commercial 
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services;  interdependency  among  services  driven  by  changing  business  practices  and  applied 
information  technology;  and  vulnerabilities  to  failures  and  disruptions  from  asymmetric  attacks, 
malicious  intrusion,  inadvertent  error,  and  natural  disasters. 

E.  Peacetime  Engagement.  Chair:  Bruce  Powers.  This  working  group  will  focus  on  the 
peacetime  presence  of  US  forces  throughout  the  globe  as  a  means  to  safeguard  our  vital  national 
interests,  promote  regional  stability,  deter  potential  adversaries,  and  build  cooperative  relations 
with  the  world’s  most  influential  countries.  In  maintaining  a  substantial  overseas  presence,  we 
will  maintain  the  capability  to  conduct  the  full  range  of  military  operations  through  a  combination 
of  permanently  stationed  forces,  rotationally  deployed  forces,  temporarily  deployed  forces,  and 
infrastructure. 

F.  Information  Architectures.  Chair:  Patsy  McGrady.  This  working  group  will  focus  on  the 
implications  of  information  architecture  (system,  technical,  and  operational)  for  the  analysis  of 
C4ISR.  Included  in  this  discussion  will  be  information  management  (process)  and  infrastructure. 
Critical  questions  include  how  can  analysis  contribute  to  developing  information  management 
architectures  and  doctrines  that  will  facilitate  getting  the  right  information  to  the  right  place  at  the 
right  time  and  can  analysis  be  flexible  enough  to  evaluate  alternative  processes. 

G.  Analytical  Techniques  and  Tools.  Co-chairs:  Dr.  Alfred  Brandstein  and  Dr.  Roy  Rice. 

This  working  group  will  focus  on  the  leading  edge  of  analysis,  specifically  the  latest 
developments  in  modeling  (e.g.,  genetic  algorithms)  and  their  applicability  to  the  analysis  of 
C4ISR.  Define  the  appropriate  use  of  various  methodological  constructs  (process,  case-based, 
network-oriented,  etc.)  to  conduct  C4ISR  analysis.  Address  the  adequacy  of  supporting 
algorithms,  simulations,  experiments,  and  exercises. 

H.  Synthesis  Working  Group.  Chair:  Dr.  Stuart  Starr.  This  working  group  will  take  a  broader 
view,  determine  any  commonality  of  issues  and  concerns  across  the  spectrum  of  working  groups, 
and  provide  an  integrating  perspective.  Two  specific  issues  of  interest  for  this  group  will  be  the 
utility  of  the  Joint  Warfare  System  (JWARS)  and  the  important  of  information  dominance. 

Tasking.  Working  groups  will  be  directed  to  address  the  following  questions  and  concerns  in  the 
following  priority  order; 

Characterize  C4ISR  within  the  working  group  focus  area.  Structure  and  decompose  the 
elements  of  C4ISR.  Describe  the  scenario  space,  key  factors,  and  concerns  that 
potentially  affect  the  analysis  and  representation  of  C4ISR.  Comment  on  the  adequacy  of 
current  mission  area  CONORS  to  support  assessment  of  the  C4ISR  contribution  in  that 
area. 


Define  the  relative  worth  of  C4ISR.  What  are  the  relative  contributions  of  C4ISR 
(information),  firepower  (weapons  and  platforms),  and  force  employment  strategies  (e.g.. 
Dominant  Maneuver)?  How  can  analysts  make  meaningful  tradeoff  analyses?  What  are 
the  breakpoints  in  C4ISR  capability  and  spread? 

Develop  and  recommend  Measmes  of  Merit.  Characterize  and  organize  —  for  primary 
(warfare)  and  secondary  (C4ISR)  functions  —  the  applicable  measures,  metrics,  and 
variables  including  Measures  of  Performance  (MOPs),  Measures  of  Effectiveness 
(MOEs),  Measures  of  Force  Effectiveness  (MOFEs),  etc.  Assess  how  well  these  metrics 
can  establish  the  value  of  C4ISR  relative  to  other  contributors  to  force  effectiveness. 
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Evaluate  how  current-or  new-metrics  can  be  used  to  calculate  Return  on  Investment 
(ROI)forC4ISR. 


Identify  and  describe  tools.  Identify  and  describe  models,  simulations,  decision  support 
systems,  and  other  C4ISR-applicable  software  tools  in  terms  of  capability,  status, 
effectiveness,  and  ability  to  contribute  to  the  development  of  operational  and  technical 
architectures.  Identify  needs  for  new  software  tools  to  support  advanced  C4ISR  analysis. 

Analyze,  synthesize,  and  infer.  Identify  common  issues  and  concerns  across  each  of  the 
four  topic  areas  and  draw  generalized  conclusions  and  make  relevant  recommendations. 

Sequence  of  Events.  The  morning  of  the  first  day  will  be  devoted  to  a  plenary  session  for 
speakers.  The  speakers  will  provide  differing  perspectives  on  the  problem  of  C4ISR  analysis.  The 
keynote  speaker  will  be  asked  to  provide  the  perspective  of  the  decision  maker  as  the  ultimate 
customer  of  analytical  products.  The  second  speaker(s)  will  provide  an  operational  perspective, 
specifically  the  view  of  various  CINCs.  The  last  speaker  will  provide  a  technical  perspective.  At 
the  end  of  these  sessions,  the  Chair  will  provide  a  charge  to  the  working  groups  to  ensure  all 
participants  share  a  common  vision  of  the  products  required.  The  afternoon  will  be  devoted  to 
working  group  sessions  under  the  supervision  of  the  working  group  chairs  to  begin  discussions  on 
the  designated  evaluation  issue.  At  the  end  of  this  day,  there  will  be  a  social  event  (mixer).  The 
second  day  will  be  devoted  to  parallel  working  group  sessions  to  further  develop  their 
discussions.  There  will  be  a  luncheon  speaker  on  this  day.  The  morning  of  the  third  day  will 
consist  of  rinal  working  sessions  to  finalize  their  presentations  and  written  products,  followed  by 
a  plenary  sessions  for  working  group  presentations. 

Attendees 

Attendance  will  be  controlled  via  invitation.  Attendees  will  include  invited  experts  from  OSD,  all 
Services,  the  Joint  Staff,  Federally  Funded  Research  and  Development  Centers,  operational 
commanders,  and  DoD  contractors.  Workshop  chairs  will  control  membership  of  their  sessions  in 
conjunction  with  the  Organizing  Committee.  Attendance  will  be  limited  to  150  people. 

Products 

There  will  be  four  specific  products  generated  as  a  result  of  this  workshop: 

•  A  scripted  briefing  for  the  MORS  Sponsors  addressing  the  workshop  objectives,  findings, 
conclusions,  and  recommendations. 

•  A  proceedings  containing  summaries  of  all  sessions  and  copies  of  appropriate  briefing 
slides  and  presentations. 

•  A  general  session  presentation  for  the  67th  MORSS. 

•  h  PHALANX  ^n\c\c 

Proponents 

Proponents  for  the  workshop  are: 

Army  Deputy  Under  Secretary  of  the  Army  (Operations  Research)  (DUSA(OR)) 

Navy  Director  of  Program  Resource  Appraisal,  Chief  of  Naval  Operations  (N81) 

Director  for  Command  and  Control,  DCS,  Air  and  Space  Operations,  HQUSAF 
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Planning  and  Organizing  Committee 

General  Chair:  RADM  Robert  Nutwell,  USN 

Deputy  Director 

Space,  Information  Warfare  Command  and  Control 
Directorate  (N6B) 

Chief  of  Naval  Operations 
2000  Navy  Pentagon 
Washington,  DC  20350-2000 
(703)  601-1200 

Technical  Chair:  Dr.  Russ  Richards 

The  MURE  Co 
1820  Dolley  Madison  Blvd 
McLean,  VA  22102 
(703)  883-6787 

Deputy  Chair:  CAPT  Jay  Kistler,  USN 

Director,  Navy  Modeling  and  Simulation 
Management  Office  (N6M) 

Chief  of  Naval  Operations 
2000  Navy  Pentagon 
Washington,  DC  20350-2000 
(703)  601-1482 

Synthesis  Chair:  Dr.  Stuart  Starr 

The  MURE  Co 
1820  Dolley  Madison  Blvd 
McLean,  VA  22102-3481 
(703)  883-5494 

Site  Coordinator:  LTC  Jerry  Nye,  USA 

U.S.  Army  War  College 
Carlisle  Barracks 
Carlisle,  PA  17013-5099 
(717)  245-4353 

Administrative  Coordinators:  Richard  Wiles  and  Natalie  Addison 

MORS,  Suite  202 
101  S.  Whiting  Street 
Alexandria,  VA  22304 
(703)751-7290 

Organizing  Committee:  Army  -  TBA 

Air  Force  -  Cla)don  J.  Thomas,  FS,  AFSAA 

Marines  -  Dr.  Alfred  Brandstein 

OSD  -  Dr.  Jackie  Henningsen,  FS 

Joint  Staff  -  Peter  Byrne 

NRO-TBA 

DISA-TBA 
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At  large: 


Dr.  Dave  Alberts  (ASD(C3I)) 
Dr.TomCedel(TASC) 
Chattier  Chris  (OSD) 

Barry  Dysart  (N6M  -  SAIC) 
Dean  Free  (N81) 

Paul  Hobbes  (N81  -  SAIC) 
Susan  Iwanski  (SPA) 

Dr.  Jerry  Kotchka,  Boeing 
Dr.  Ivar  Oswalt  (Kapos) 

Vince  Roske  (J-8) 

John  Kelsey  (TASC) 

Doug  Sizelove  (ODIJSA(OR)) 
Dr.  Larry  Wiener  (N6C5) 

LTC  Patrick  Vye,  USA  (J-6) 


Administration 

Name:  Analysis  of  C4ISR  2010 
Dates:  27-29  October  1998 

Location:  Center  for  Strategic  Leadership,  Carlisle  Barracks,  PA 
Fee:  Federal  Government  Employees  $175;  All  others  $350 
Attendance:  Limited  to  150 
Classification:  SECRET 
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Agenda 

Tuesday  -  27  October 


0700 

Registration  and  Continental  Breakfast 

Bliss  Hall 

0800 

MORS  President’s  Welcome 

Denny  Baer 

0810 

Welcome  by  Commandant 

Army  War  College 

MG  Robert  Scales,  USA 

0820 

Overview  of  Workshop 

Introduction  of  Keynote  Speaker 

RADM  Nutwell,  USN 

0840 

Keynote: 

Mr.  Art  Money,  ASD(C3I) 

"The  Decision  Maker’s  Perspective" 

0930 

"The  Operators  Perspective" 

TEA 

1030 

Break 

“C3I  The  Past,  Present  and  Future” 

Dr.  Stuart  Starr,  MITRE 

1145 

Transportation  from  Bliss  Hall  to  Collins  Hall 

1200 

Catered  Lunch 

Collins  Hall 

1300 

Working  Group  Sessions 

1430 

Break 

1700 

Mixer 

Letort  View 
Comm.  Center 

Wednesday  -  28  October 

0715 

Continental  Breakfast 

Collins  Hall 

0800 

Working  Group  Discussions 

1145 

Transportation  from  Collins  Hall  to  Letort  View  Community  Center 

1200 

Buffet  Lunch  With  Guest  Speaker: 

"Federated  analysis  of  C4ISR  in  Warfare" 

Mr. Charles  Taylor, 

Letort  View 

OASD(C3I)/DSC 

Comm.  Center 

1315 

Transportation  from  Letort  View  Community  Center  to  Collins  Hall 

Collins  Hall 

1330 

Working  Group  Sessions 

1430 

Break 

1700 

Adjourn 

Thursday  -  29  October 

0715 

Continental  Breakfast 

Collins  Hall 

0800 

Working  Group:  Prepare  Briefings  to 

Summarize  Deliberations 

1000 

Plenary  Session:  "Blackhawk  Down" 

Mark  Bowden 

Letort  View 

1200 

Catered  Lunch 

Conun.  Center 

1300 

Plenary  Session:  Working  Group  Reports 

WG  Chairs 

Bliss  Hall 

1315 

Major  Theater  War 

1330 

Smaller  Scale  Contingencies 

1345 

Operations  Other  Than  War 

1400 

Peacetime  Engagement 

1415 

Infrastructure  Assurance 

1430 

Break 

1445 

Information  Architectures 

1500 

Analytical  Techniques  and  Tools 

1515 

Synthesis 

1530 

Closing  Remarks 

RADM  Nutwell 

1600 

Adjourn  Workshop 
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Milestone  Plan 


Date 

"D" 

Task 

Responsibility 

30  Jan 

-270 

Initiate  draft  TOR  procedure 

Initiator 

31  Mar 

-210 

Select  tentative  dates 

Initiator,  MORS 

30  Apr 

-180 

Provide  "For  Conunent"  draft  of  TOR  to  Sponsors, 
VP(MO)  and  other  interested  organizations  and 
individuals  for  review 

MORS  office 

30  May 

-150 

Revise  TOR 

Initiator,  MORS  office 

9  Jun 

-140 

Circulate  initial  draft  TOR  to  MORS  Office  and 
committee  for  comments 

Chair 

Solicit  candidate  for  working  group  chairs 

29  Jun 

-120 

Circulate  final  draft  TOR  to  MORS  Office  and 
proponents  for  concurrence  and  to  other  Sponsors 
and  organizations  for  information 

MORS  office 

29  Jul 

-90 

Approve  TOR,  program  chair,  budget  and  fees 

Executive  Council 

Select  working  group  chairs 

Chair 

3  Aug 

-85 

Mail  ACP,  if  appropriate 

MORS  office 

8  Aug 

-80 

Select  organization  to  be  invited  and 
prepare  letter  inviting  nominations 

Chair 

13  Aug 

-75 

Mail  invitations  to  nominating  organizations 

Chair 

28  Aug 

-60 

Select  read-ahead  material 

Chair 

7  Sep 

Select  nominations  or  requests  for  applications 

Nominating  orgs. 

-50 

Select  invitees 

Committee 

12  Sep 

-45 

Assign  nominees  to  working  groups 

Committee 

Mail  invitations  and  working  group  assignments 

MORS  office 

Provide  read-ahead  materials  and  releases  to 

MORS  office 

Committee 

17  Sep 

-40 

Mail  read-ahead  materials 

MORS  office 

2  Oct 

-25 

Pre-registration,  security  clearances 
due  to  MORS  office 

Invitees 

27- 29  Oct 

Conduct  workshop 

Chair/Committee 

26  Nov 

30 

Brief  Sponsors 

Chair 

26  Dec 

60 

Submit  After- Action  Report 

Chair 

25  Jan 

90 

Complete  written  products 

Committee 

24  Feb 

120 

Approve  written  products 

Publications  committee 

26  Mar 

150 

Review  approved  products 

Proponents 

25  Apr 

180 

Distribute  approved  products 

MORS  office 

1.  28  September  1998 
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Ms  Natalie  S  Addison 
MORS,  Suite  202 
101  S.  Whiting  Street 
Alexandria  VA  22304 
TELEPHONE: 

OFF:  DSN:  FAX:  703-751-8171 
Email:morsvpa@aol.com 

DR  David  S  Alberts 
OASD  (C3I) 

CG2,  Suite  910 
1225  Jefferson  Davis  Hwy 
Arlington  VA  22202 
TELEPHONE: 

OFF:  703-287-0317  DSN:  FAX:  ((703) 
Email:David.Alberts@osdpentagon.mil 

DR  Patrick  D  Allen 
Cubic  AppKcations  Inc 
4550  3rdAveSE 
Olympia  WA  98503 
TELEPHONE: 

OFF:  (360)-438-6078  DSN:  FAX:  (360)-493-6195 
Email:pat.allen@cubicxom 

Ms.  Carol  Anway-Wiese 
Boeing,  Phantom  Works 
Warfare  Analysis,  MC  8R-44 
PO  Box  3999 
Seattle  WA  98124-2499 
TELEPHONE: 

OFF:  253-773-7574  DSN:  FAX:  253-773-7233 

Mr.  Edgar  D  Arendt 

US  Army  TRAC 

255  Sedgwick  Ave 

Fort  Leavenworth  KS  66027-2345 

TELEPHONE: 

OFF:  913-684-9215  DSN:  552-9215  FAX:  913-684-9232 

MR  Dennis  R  Baer 
Logicon 

2100  Washington  Blvd 
Arlington  VA  22204-5703 
TELEPHONE: 

OFF:  (703)-312-2149  DSN:  FAX:  (703)-312-2780 
Email:  drbaer@aoLcom  or  dbaer@logicon.com 


MR  Timothy}.  Bailey 

US  Army  TRAC  Andysis  Center 

Attn:  ATRC-SAS 

255  Sedgwick  Ave 

Fort  Leavenworth  KS  66027-2345 

TELEPHONE: 

OFF:  (913)-684-9205  DSN:  552-  FAX:  (913)-684-9191 
Email:baileyt@trac.army.mil 

MAJ  James  B  Barnes 
AFSAA 

1570  Air  Force  Pentagon 
Washington  DC  20330-1570 
TELEPHONE: 

OFF:  703-588-8679  DSN:  425-8679  FAX:  703-425-2220 

Email:james.bames@pentagon.af.mil 

MR.  Stephen  F  Barnett 
National  Security  Agency 
STE  6681 
9800  Savage  Rd 
Ft  Meade  MD  20755-6681 
TELEPHONE: 

OFF:  301-688-2837  DSN:  FAX:  301-688-2311 

MS  Beverly  D  Barnette 

US  Army  Research  Laboratory 

Human  Research  &  Engineering  Directorate 

Attn:  AMSRL-HR-SA 

Aberdeen  Proving  Ground  MD  21005 

TELEPHONE: 

OFF:  (410)-278-5856  DSN:  298-5856  FAX:  (410)-278- 
3620 

Email:dbamett@ARL.MIL 

Mr.  Stephen  A  Beardslee 

Boeing 

MC  8R-44 

PO  Box  3999 

Seattle  WA  98124-2499 

TELEPHONE: 

OFF:  253-773-2405  DSN:  FAX:  253-773-7233 
Email:  stephen.a.beardslee@boeing.com 
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Mr.  Donald  R  Bennett 
HQDA,  Director  of  Military  Support 
Critical  Asset  Assurance  Program 
Army  Operations  Center  Pentagon 
Washington  DC  20310 
TELEPHONE: 

OFF:  703-697-1380  DSN:  227  FAX:  703-697-3147 
MAJ  Bruce  S  Bishop 

Air  Force  Studies  and  Analysis  Agency /SAAI 

AFSAA/SAAI 

1570  Air  Force  Pentagon 

Washington  DC  20330-1570 

TELEPHONE: 

OFF:  703-588-8606  DSN:  425-8606  FAX:  703-588-2220 
Email:bruce.bishop@pentagon.a£mil 

DR  Joseph}  Bolmarcich 
QUANTICS  Inc 
801  Springdale  Dr  #120 
Exton  PA  19341-2866 
TELEPHONE: 

OFF:  (610)-993-9993  DSN:  FAX:  (610)-524-7429 
Email:joe@quantics.com 

MS  Mary  T  Bonnet 
AFSAA/SAJ 

1570  Air  Force  Pentagon  #1E386 
Washington  DC  20330-1570 

OFF:  703-588-6910  DSN:  425-6910  FAX:  703-588-8779 
Email:M[ary.Bonnet@pentagon.af.mil 

LCDR  Randall  G  Bowdish 
OPNAV  (N513) 

Room  4A562 
2000  Navy  Pentagon 
Washington  DC  20350-2000 
TELEPHONE: 

OFF:  703-693-7000  DSN:  223-7000  FAX:  703-614-5530 

Mr  Thomas  E  Bozek 
OASD  (C3I) 

Room  1E670 

6000  Defense  Pentagon 

Washington  DC  20301-6000 

OFF:  703-697-3215  DSN:  FAX:  703-697-9304 

Email:  tom.bo2ek@os  d.pentagon.mil 


DR  Alfred  G  Brandstein 

Marine  Corps  Combat  Development  Command 

Studies  and  Analysis  Division 

3300  Russell  Road 

QuanticoVA  22134 

TELEPHONE: 

OFF:  (703)-784-3235  DSN:  278-  FAX:  (703)-784-3547 
Email:algebra@quantico.usmc.mil 

MR  Kevin  Brandt 
The  MITRE  Corporation 
W159  Joint  Warfighting  Ctr 
POB  51190 

Fort  Monroe  VA  23651-0190 

OFF:  (757)-723-2460  DSN:  680-5360  FAX:  (757)-723- 

4785 

Email:brandtk@mitre.org 

Mr  David  W  Cann 
NUWC  Division  Newport 
Code  63  Bldg  1320/5 
1176  Howell  Street 
Newport  RI  02841 

OFF:  401-832-5733  DSN:  920-5733  FAX:  401-832-7440 
Email:  canndw@npt.nuwc.navy.mil 

LTC  Patrick  M  Cannon 
JADS  Joint  Test  Force 
11104  MenaulBlvdNE 
Albuquerque  NM  87112 
TELEPHONE: 

OFF:  (505)-846-1385  DSN:  246-  FAX:  (505)-846-0406 
Email:cannonp@jads.kirdand.af.mil 

CAPT  Robert  S  Carnes 

Marine  Corps  Warfighting  Laboratory 

3255  Meyers  Avenue 

Quantico  VA  22134 

TELEPHONE: 

OFF:  DSN:  FAX: 

Email: 

CAPT  Steve  Chapman 

Chief  of  Naval  Operations,  N6M 

Director,  Navy  M&S  Management  Office 

2000  Navy  Pentagon 

Washington  DC  20350-2000 

TELEPHONE: 

OFF:  703-601-1482  DSN:  329-1482  FAX:  703-601-1337 
Email:chapman.steve@hq.navy.mil 


K-2 


MR  E.  Christopher  Chartier 

ASD(C3I)  Joint  C4ISR  Decision  Support 

Analysis 

Crystal  Mall  Three,  6th  Floor 
1931  Jefferson  Davis,  HWY 
Arlington  VA  22202 
TELEPHONE: 

OFF:  (703)  607-0632  DSN:  327-  FAX:  (703)  607-0603 
Email:christopher.chartier@osd.niil 

MR  George  T  Cherolis 

TRW  (Theater  Air  C2  Simulation  Facility) 

TACCSF 

6001  Indian  School  NE 
Albuquerque  NM  87110 
TELEPHONE: 

OFF:  (505)-846-4474  DSN:  246-  FAX:  (505)-846-1872 
Email:gcheroli@taccsf.kirdand.af.mil 

MR  Denis  T  Clements 
OSD  (PA&E)  GRCI 
1900  Gallows  Road 
Vienna  VA  22182 
TELEPHONE: 

OFF:  703-696-9490  DSN:  FAX:  703-556-4261 
Email:dclements@grci.com 

MR  Gary  Q  Coe 
Institute  for  Defense  Analyses 
1801  N  Beauregard  Street 
Alexandria  VA  22311 
TELEPHONE: 

OFF:  703-845-6628  DSN:  FAX:  703-845-6809 
Email:gcoe@ida.org 

COL  William  J  Cooper 
Joint  Staff,  J6E 
The  Pentagon 

Washington  DC  20318-6000 

OFF:  703-693-3492  DSN:  223-  FAX:  703-614-5291 

Prof  Thomas  J  Czerwinski 
National  Defense  University 
IRMC  MarshaU  HaU 
Fort  McNair  VA  20319-5066 
TELEPHONE: 

OFF:  202-685-2245  DSN:  325-2245  FAX:  202-685-3664 


MAJ  David  A  Davies 

ACC/XP-SASF 

STE202 

204  Dodd  Blvd 

Langley  AFB  VA  23665 

TELEPHONE: 

OFF:  757-764-5330  DSN:  574-5330  FAX:  757-764-7217 
Email:  david.  davies@langley.af  mil 

CAPT  Lawrence  L  Dick 
OP  NAV 

Director  of  Navy  Training/ Acquisition  Div 
(N75) 

2000  Navy  Pentagon 
Washington  DC  20350-2000 
TELEPHONE: 

OFF:  703-697-0182  DSN:  224-1098  FAX:  703-695-5698 
Ms  Kathy  R  Dunn 

Office  of  Defenes  Research  &  Engineering 
Information  Technologies 
3080  Defense  Pentagon,  Room  3D367 
Washington  DC  20301-3080 
TELEPHONE: 

OFF:  703-614-0207  DSN:  FAX:  703-693-2981 
Email:dunnkr@acq.osd.mil 

CAPT  Robert  W  Eberth  USNR 

OPNAV  N85/Marine  Corps  Warfighting  Lab 

2006  Hawkins  Ave 

Quantico  VA  22134 

TELEPHONE: 

OFF:  703-784-3785  DSN:  278-3785  FAX:  703-784-4421 
Email:eberthr@mcwl  .quantico.usmc.mil 

Dr  R.  Evan  Ellis 
SAIC 
MS  T2-6-9 
1710  Goodridge  Dr. 

McLean  VA  22102 

OFF:  703-748-5345  DSN:  FAX:  703-556-3364 
Email:r.evan.el]is@cpmx.saic.com 

MR  Olaf  L  Elton 
MITRE 
JWFL,  Bldg  96 
Ft  Monroe  VA  23651 
TELEPHONE: 

OFF:  757-723-2460  DSN:  680-4257  FAX  757-723-4785 
Email:elton@mitre.org 
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Mr.  Roland  P  Fabia 
DIA  (Attn:  TWP) 

Washington  DC  20340-5100 
TELEPHONE: 

OFF:  202-231-4307  DSN:  428-4307  FAX:  202-231-8169 
Email:affabrp@dia.osis.gov 

Mr.  Richard  E  Favela 
Betac  Corporation 
2001  N.Beauregard  St 
Alexandria  VA  22311 
TELEPHONE: 

OFF:  703-824-3100  DSN:  FAX  703-824-0333 
Email:favela@betac.com 

MR  John  R  Ferguson 
SAIC 

Crystal  Square  5,  Suite  202 
1755  Jefferson  Davis  Hwy 
Arlington  VA  22202 
TELEPHONE: 

OFF:  (703)-414-0193  DSN:  FAX:  (703)-413-5116 
Email:fergusonj@mail.etas.com 

Mr.  David  P  Fichtner 

NRO 

(23F02f^ 

14675  Lee  Road 
ChantiUyVA  20151-1715 
TELEPHONE: 

OFF:  703-808-5696  DSN:  FAX:  703-808-5789 

MAJ  Steven  L.  Forsythe 

AFSAA 

RM  1D377 

1570  Air  Force  Pentagon 
Washington  DC  20330-1570 
TELEPHONE: 

OFF:  703-588-8292  DSN:  425-8292  FAX:  703-588-0220 
Email:steven.forsyth@pentagon.af.niil 

MR  W.  Dean  Free 

Chief  of  Naval  Operations  (N816) 

2000  Navy  Pentagon 
Washington  DC  20350-2000 
TELEPHONE: 

OFF:  (703)  697-0679  DSN:  227-0679  FAX:  (703)-697- 
0738 

Email:  free.dean@hq.navy.niil 


MR  John  S  Furman 
The  MITRE  Corporation 
MSW647 

1820  Dolley  Madison  Blvd 
McLean  VA  22102 

OFF:  (703)-883-6342  DSN:  FAX:  (703)-883-1370 
Email:None 

Dr  Donald  P  Gaver  Jr. 

Naval  Postgraduate  School 
Dept  of  OR 
Monterey  CA  93943 
TELEPHONE: 

OFF:  (408)-656-2605  DSN:  878-  FAX:  (408).656-2595 
Email:dgaver@nps.navy.mil 

Mr.  John  T.H.  Germanos 

MTMCTEA 

STE130 

720  Thimble  Shoals  Blvd 
Newport  News  VA  23606-2574 
TELEPHONE: 

OFF:  757-878-4313  DSN:  927-4313  FAX  757-599-1560 
Email: 

COL  Richard  B  Gibson 
ESC/DIS 
5  Eglin  St 

Hanscom  AFB  MA  01731-2100 
TELEPHONE: 

OFF:  781-377-6554  DSN:  478-6554  FAX:  781-377-1153 
Email:gibsonh@hansom.af.mil 

MR  Lawrence  A  GUcoes 
Logistics  Management  Institute 
2000  Corporate  Ridge  Rd 
McLean  VA  22102-7805 
TELEPHONE: 

OFF:  (703)-917-7574  DSN:  FAX  (703)-917-7594 
EmaiI:lglicoes(@kni.org 

Mr  Charles  R  Hall  III 
The  MITRE  Corporation 
MSW525 

1820  DoUey  Madison  Blvd 
McLean  VA  22102-3481 
TELEPHONE: 

OFF:  703  883-6260  DSN:  FAX  703  883-7852 
Email:  chall@mail04.mitre.org 
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MR  Robert  G  Harding 

Chief  of  Naval  Operations  (N812D) 

N816D4 

2000  Navy  Pentagon 
Washington  DC  20350-2000 
TELEPHONE: 

OFF:  703-697-0463  DSN:  227-0463  FAX:  703-697-0738 
Email:hartling.bob@hq.navy.mil 

Mr.  David  L  Hawk 
Scientific  Research  Corporation 
PO  Box  18076 

Albuquerque  NM  87185-8076 
TELEPHONE: 

OFF:  505-846-4474  DSN:  246-4474  FAX:  505-846-1872 
Email:  dhawk@taccsfkirtiand.af.mil 

DR  Richard  E  Hayes 
Evidence  Based  Research,  Inc 
STE250 

1595  Spring  Hill  Road 
Vienna  VA  22182-2228 
TELEPHONE: 

OFF:  (703)-893-6800  DSN:  FAX  (703)-821-7742 
Email:rehayes@ebrinc.com 

LtCol  Kenneth  R  Hebert 
JADSJTF 

11104  Menaul  Blvd  NE 
Albuquerque  NM  87112-2454 
TELEPHONE: 

OFF:  505-846-0967  DSN:  246-0967  FAX: 

Ms  Dianna  L  Heitman 

Tracor  Systems  Technologies,  Inc 

JPO-STC 

PO  Box  381 

DahlgrenVA  22448 

OFF:  540-653-7660  DSN:  FAX:  540-653-2522 

DR  Jacqueline  R.  Henningsen  FS 

HQUSAFAXOC 

1480  Air  Force  Pentagon 

Washington  DC  20330-1480 

TELEPHONE: 

OFF:  703-695-1833  DSN:  225-1833  FAX:  703-695-5758 
Email:henningsenj@pentagon.afmil 

MAJ  Anthony  R  Herlihy 
Marine  Security  Guard  Battalion 
Operations  Officer  (S3) 

2007EmotRd 
Quantico  VA  22134 
TELEPHONE: 

OFF:  703-784-2142  DSN:  FAX: 
Email:herlihya@quantico.usmc.mil 


Mr.  Frederick  W  Herr 

National  Communications  System 

N5 

701  S.  Court  House  Road 
FaUs  Church  VA  22043 
TELEPHONE: 

OFF:  703-607-6220  DSN:  FAX:  703-607-4826 

MS  Virginia  Susan  Hudson 

Naval  Surface  Warfare  Center 

Susan  Hudson,  Dahlgren  Division  QOT) 

17320  Dahlgren  Road 
Dahlgren  VA  22448-5100 
TELEPHONE: 

OFF:  540-653-8730  DSN:  249-8730  FAX:  540-653-4977 

MS  Susan  M  Iwanski 
Systems  Planning  &  Analysis 
Suite  400 

2000  N  Beauregard  Street 
Alexandria  VA  22311-1712 
TELEPHONE: 

OFF:  (703)  578-5636  DSN:  FAX:  (703)  931-9254 
Email:siwanski@spa.com 

Mr.  Lawrence  Karl  Jaffe 
Betac  Corporation 

2001  N  Beauregard  St 
Alexandria  VA  22311 
TELEPHONE: 

OFF:  703-824-3100  DSN:  FAX:  703-824-0333 

MR  Norman  M  Johnson 
Northrop  Grumman 
Aircraft  Systems  Div,  M/S  C63-05 
South  Oyster  Bay  Rd 
Bethpage  NY  11714 
TELEPHONE: 

OFF:  (516)  575-1878  DSN:  FAX:  (516)  346-9740 
Email:norm_johnson@atdc.northgrum.com 

MR  John  S.  Kelsey 

TASC 

Suite  1500 

1101  Wilson  Blvd 

Arlington  VA  22209 

TELEPHONE: 

OFF:  703-558-7400  DSN:  FAX:  (703)-524-6666 

Email:jskelsey@tasc.com 
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PROF  WiUiam  G  Kemple 
Naval  Postgraduate  School 
C3  Academic  Group 
589  Dryer,  Room  200A 
Monterey  CA  93943 
TELEPHONE: 

OFF:  (831)-656-3309  DSN:  878-3309  FAX:  (831)-656- 
3679 

Email:kemple@nps.navy.mil 

LCDR  Martin  P  Keutel 
OPNAVN6 

Chief  of  Naval  Operations 
2000  Navy  Pentagon 
Washington  DC  20350-2000 
TELEPHONE: 

OFF:  703-601-1487  DSN:  329-1487  FAX:  703-601-1353 

Mr.  Joseph  Keith  Kirkland 
Evidence  Based  Research,  Inc 
1595  Spring  HiU  Road,  STE  250 
Vienna  VA  22182 
TELEPHONE: 

OFF:  703-287-0314  DSN:  FAX:  703-821-7742 

CAPTJayRKisder 
CNO  Staff  (N6M) 

2000  Navy  Pentagon 
Washington  DC  20350-2000 
TELEPHONE: 

OFF:  (703)-601-1480  DSN:  329-1480  FAX:  703-601-1337 
Email:Kisder.ja3r@hq.navy.mil 

Dr.  Stephen  R  Kolek 
MIT  Lincoln  Laboratory 
Bldg  S,  Room  S4-1 65 
244  Wood  Street 
Lexington  MA  02421-9185 
TELEPHONE: 

OFF:  781-981-4170  DSN:  FAX  781-981-7615 
Email:kolek@ll.mit.edu 

DR  Jerry  A  Kotchka 
Boeing 

Mail  Code  0641251 
PO  Box  516 

St.  Louis  MO  63166-0516 
TELEPHONE: 

OFF:  (314)-777-9990  DSN:  FAX:  (314)-232-7917 
Email:jerry.a.kotchka@boeing.com 


MRWilHamMKroshl 
Johns  Hopkins  University/APL 
Applied  Physics  Lab 
11 100  Johns  Hopkins  Road 
Laurel  MD  20723-6099 
TELEPHONE: 

OFF:  240-228-4870  DSN:  FAX:  240-228-5910 
Email:  William.Kroshl@jhapl.edu 

MS  Cynthia  Kee  LaFreniere 
Military  Operations  Research  Society 
#202 

101  S  Whiting  St 
Alexandria  VA  22304-3416 
TELEPHONE: 

OFF:  (703)-751-7290  DSN:  FAX:  (703)-751-8171 
Email:morscyn@aol.com 

LtCol  Stephen  S  Lisi 
The  Joint  Staff 

CA  Systems  Directorate,  Room  1E833 
Technology  and  Architecture  Division 
Washington  DC  20318-6000 
OFF:  DSN:  223-5332  FAX  703-697-6610 

MAJ  David  E  Lockhart 

USAMSAA 

392  Hopkins  Road 

Aberdeen  Proving  Ground  MD  21005-5071 
TELEPHONE: 

OFF:  410-278-6435  DSN:  298  FAX:  410-278-4694 

MR  Robert  Craig  MacFarlane 

SAIC 

6th  Floor 

1777  North  Kent  Street 
Arlington  VA  22209 
TELEPHONE: 

OFF:  703-588-8295  DSN:  FAX  703-588-0220 
Email:  Robert.MacFarlane@pentagon.af.mil 

MS  Marilyn  K  Macklin 
HQ  DA  ODCSINT 
Room  9302 

2511  Jefferson  Davis  Hwy 
Arlington  VA  22202 
TELEPHONE: 

OFF:  703-604-2454  DSN:  664-2454  FAX:  703-601-0553 
Email:marilyn.macklin@hqda.army.mil 
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ILT  Geoffrey  S.  Maron 

AFSAA/SAAB 

1570  Air  Force  Pentagon 

Washington  DC  20330-1570 

TELEPHONE: 

OFF:  703-588-8289  DSN:  425-8289  FAX:  703-425-0220 
Email:Geoffrey.Maron@pentagon.af.niil 

MR  Earl  Daniel  Mathis 
Naval  Surface  Warfare  Center 
Joint  Program  Office  for  STC 
17320  Dahlgren  Rd 
Dahlgren  VA  22448 
TELEPHONE: 

OFF:  540-653-1355  DSN:  249-1355  FAX:  540-653-7648 

Ms  Erin  M  McDonald 
The  Johns  Hopkins  University/ APL 
11100  Johns  Hopkins  Road 
Laurel  MD  20723-6099 

OFF:  240-228-8439  DSN:  FAX:  240-228-6583 

MS  Patsy  M  McGrady 
ASD  (C3I) 

Crystd  Mass  3,  Room  7104 
1931  Jefferson  Davis  Hwy 
Arlington  VA  22202 

OFF:  703-607-0477  DSN:  FAX:  703-607-0553 
Email:mcgrdyp@osd.pentagon.mil 

CPT  Otis  W  McGregor  III 

US  Army  Land  Information  Warfare  Acty 

STE404 

5111  Leesburg  Pike 

Falls  Church  VA  22041-3210 

TELEPHONE: 

OFF:  703-681-4355  DSN:  FAX:  703-681-3437 
Email:  owmcgre@vulcan.belvoir.army.mil 

Mr.  William  C  McIntyre  III 

HQ  SWC/AE 

STE  83 

730  Irwin  Ave 

Falcon  AFB  CO  80912-7383 
TELEPHONE: 

OFF:  719-567-9338  DSN:  560-9338  FAX:  719-567-9496 

Mr  Andrew  W  Melton 
Computer  Systems  Center  Inc 
Suite  520 

6225  Brandon  Ave 
Springfield  VA  22150-2519 
TELEPHONE: 

OFF:  703-866-4000  DSN:  FAX:  703-866-4001 
Email:amelton@csci-va.com 


MR  Sam  E  Middlebrooks 

US  Army  Research  Lab 

Attn:  AMSRL-HR-S  APG 

Aberdeen  Proving  Ground  MD  21005-5425 

TELEPHONE: 

OFF:  (410)-278-9523  DSN:  278-FAX:  (410)-278-3148 
Email:smiddleb@arl.mil 
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